[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:04:55 dist-test-slave-2d19 krb5kdc[3131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:04:55 dist-test-slave-2d19 krb5kdc[3131](info): set up 2 sockets
Aug 05 23:04:55 dist-test-slave-2d19 krb5kdc[3131](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260805 23:04:56.849929 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.328s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:04:56 dist-test-slave-2d19 krb5kdc[3131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:04:56.993108 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41697
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:41697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:04:57.130662 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:04:57.130959 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:04:57.131043 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:04:57.135047 3152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:04:57.135133 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:04:57.135174 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:04:57.135231 3152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:04:57.135268 3152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:04:57.140515 3152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:41697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:04:57.141595 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:04:57.142638 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:04:57.152185 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:57.152216 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:04:57.152232 3152 server_base.cc:1034] running on GCE node
W20260805 23:04:57.152199 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:04:57.153065 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:04:57.154181 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:04:57.155392 3152 hybrid_clock.cc:648] HybridClock initialized: now 1785971097155348 us; error 102 us; skew 500 ppm
Aug 05 23:04:57 dist-test-slave-2d19 krb5kdc[3131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:04:57.158397 3152 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:04:57.159262 3152 webserver.cc:460] Webserver started at http://127.3.12.126:34819/ using document root <none> and password file <none>
I20260805 23:04:57.159773 3152 fs_manager.cc:359] Metadata directory not provided
I20260805 23:04:57.159832 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:04:57.160013 3152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:04:57.162283 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "d0797e8044ba474883e1ad612dd4a3ef"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "01cd0252d48c108fbf3366d09423fbd3"
server_key_iv: "476c80d6efa72f434e96ea8a86867cb0"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.162874 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "d0797e8044ba474883e1ad612dd4a3ef"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "01cd0252d48c108fbf3366d09423fbd3"
server_key_iv: "476c80d6efa72f434e96ea8a86867cb0"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.166700 3152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:04:57.169236 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.170528 3152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:04:57.170676 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "d0797e8044ba474883e1ad612dd4a3ef"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "01cd0252d48c108fbf3366d09423fbd3"
server_key_iv: "476c80d6efa72f434e96ea8a86867cb0"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.170790 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:04:57.187548 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:04:57.191243 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:04:57.191464 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:04:57.198982 3152 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:41697
I20260805 23:04:57.199003 3258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:41697 every 8 connection(s)
I20260805 23:04:57.199950 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:04:57.202525 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3152
I20260805 23:04:57.202688 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:04:57.203517 3259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:04:57.204391 3121 external_mini_cluster.cc:1428] Setting key 2be72878fea63aa595194cfabe09d1f9
I20260805 23:04:57.210632 3259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef: Bootstrap starting.
I20260805 23:04:57.213446 3259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef: Neither blocks nor log segments found. Creating new log.
I20260805 23:04:57.214432 3259 log.cc:824] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef: Log is configured to *not* fsync() on all Append() calls
I20260805 23:04:57.216600 3259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef: No bootstrap required, opened a new log
I20260805 23:04:57.220808 3259 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0797e8044ba474883e1ad612dd4a3ef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41697 } }
I20260805 23:04:57.221097 3259 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:04:57.221172 3259 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0797e8044ba474883e1ad612dd4a3ef, State: Initialized, Role: FOLLOWER
I20260805 23:04:57.221678 3259 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0797e8044ba474883e1ad612dd4a3ef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41697 } }
I20260805 23:04:57.221807 3259 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:04:57.221911 3259 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:04:57.222055 3259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:04:57.222985 3259 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0797e8044ba474883e1ad612dd4a3ef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41697 } }
I20260805 23:04:57.223363 3259 leader_election.cc:302] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0797e8044ba474883e1ad612dd4a3ef; no voters:
Aug 05 23:04:57 dist-test-slave-2d19 krb5kdc[3131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:04:57.223779 3259 leader_election.cc:288] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:04:57.223888 3268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:04:57.224143 3268 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [term 1 LEADER]: Becoming Leader. State: Replica: d0797e8044ba474883e1ad612dd4a3ef, State: Running, Role: LEADER
I20260805 23:04:57.224493 3268 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0797e8044ba474883e1ad612dd4a3ef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41697 } }
I20260805 23:04:57.225389 3259 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:04:57.226411 3269 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0797e8044ba474883e1ad612dd4a3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0797e8044ba474883e1ad612dd4a3ef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41697 } } }
I20260805 23:04:57.226560 3269 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [sys.catalog]: This master's current role is: LEADER
I20260805 23:04:57.226989 3270 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0797e8044ba474883e1ad612dd4a3ef. Latest consensus state: current_term: 1 leader_uuid: "d0797e8044ba474883e1ad612dd4a3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0797e8044ba474883e1ad612dd4a3ef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41697 } } }
I20260805 23:04:57.227717 3278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:04:57.227751 3270 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef [sys.catalog]: This master's current role is: LEADER
I20260805 23:04:57.230862 3278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:04:57.231344 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:57.208228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.3.12.126:41697)
0805 23:04:57.208744 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0805 23:04:57.208767 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0805 23:04:57.209858 (+ 1091us) server_negotiation.cc:407] Connection header received
0805 23:04:57.213172 (+ 3314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:04:57.213193 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:04:57.213602 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:04:57.213958 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:04:57.216124 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.217610 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.218818 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.219656 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.219828 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:57.224540 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:04:57.224546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:04:57.224561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:04:57.224596 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:04:57.226897 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:57.227622 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:57.227628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:57.227634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:57.227785 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:57.228340 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:57.228344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:57.228348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:57.228877 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:04:57.229095 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0805 23:04:57.230009 (+ 914us) server_negotiation.cc:299] Negotiation successful
0805 23:04:57.230332 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":155,"threads_started":1}
I20260805 23:04:57.240021 3278 catalog_manager.cc:1347] Generated new cluster ID: b7eeee86094b4548ae34e23c20db1100
I20260805 23:04:57.240104 3278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:04:57.251276 3278 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:04:57.252193 3278 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:04:57.256801 3278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0797e8044ba474883e1ad612dd4a3ef: Generated new TSK 0
I20260805 23:04:57.257459 3278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:04:57.297369 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41697
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:04:57.422869 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:04:57.423135 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:04:57.423179 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:04:57.427330 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:04:57.427395 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:04:57.427472 3298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:04:57.432758 3298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:04:57.433842 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:04:57.434928 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:04:57.444944 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:57.444950 3308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:57.445032 3307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:04:57.445370 3298 server_base.cc:1034] running on GCE node
I20260805 23:04:57.445808 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:04:57.446463 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:04:57.447659 3298 hybrid_clock.cc:648] HybridClock initialized: now 1785971097447642 us; error 41 us; skew 500 ppm
Aug 05 23:04:57 dist-test-slave-2d19 krb5kdc[3131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:04:57.450519 3298 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:04:57.451371 3298 webserver.cc:460] Webserver started at http://127.3.12.65:37387/ using document root <none> and password file <none>
I20260805 23:04:57.451882 3298 fs_manager.cc:359] Metadata directory not provided
I20260805 23:04:57.451956 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:04:57.452128 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:04:57.454159 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "dfa6cb9c779741799106c7d86f849637"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "dea2149dccb5557462a25e67ba6004f6"
server_key_iv: "a028349bdfa91b8202d6eb25840c5507"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.454659 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "dfa6cb9c779741799106c7d86f849637"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "dea2149dccb5557462a25e67ba6004f6"
server_key_iv: "a028349bdfa91b8202d6eb25840c5507"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.458482 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260805 23:04:57.461023 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.462477 3298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260805 23:04:57.462634 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "dfa6cb9c779741799106c7d86f849637"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "dea2149dccb5557462a25e67ba6004f6"
server_key_iv: "a028349bdfa91b8202d6eb25840c5507"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.462751 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:04:57.478754 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:04:57.482484 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:04:57.482712 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:04:57.483394 3298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:04:57.484436 3298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:04:57.484491 3298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.484563 3298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:04:57.484599 3298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.496757 3298 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:39917
I20260805 23:04:57.496824 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:39917 every 8 connection(s)
I20260805 23:04:57.497728 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:04:57.505229 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3298
I20260805 23:04:57.505319 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:04:57.505594 3121 external_mini_cluster.cc:1428] Setting key f4883eb7e69f7f5e4888744d904a2edc
Aug 05 23:04:57 dist-test-slave-2d19 krb5kdc[3131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:04:57.512857 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:57.499705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:43281 (local address 127.3.12.126:41697)
0805 23:04:57.499845 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0805 23:04:57.499849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:04:57.500892 (+ 1043us) server_negotiation.cc:407] Connection header received
0805 23:04:57.502468 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:04:57.502471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:04:57.502520 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:04:57.502599 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:04:57.503874 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.504503 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.505739 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.506017 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.506125 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:57.508903 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:04:57.508909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:04:57.508915 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:04:57.508958 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:04:57.510469 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:57.511171 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:57.511174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:57.511175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:57.511256 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:57.511668 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:57.511671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:57.511672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:57.511845 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:04:57.511942 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0805 23:04:57.512627 (+ 685us) server_negotiation.cc:299] Negotiation successful
0805 23:04:57.512733 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:04:57.514144 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:57.500116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41697 (local address 127.3.12.65:43281)
0805 23:04:57.500707 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0805 23:04:57.500772 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0805 23:04:57.502241 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:04:57.502761 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:04:57.502768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:04:57.503200 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:04:57.503731 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:04:57.503745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.504618 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.504621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:04:57.505608 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:04:57.505618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.506197 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.506204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:04:57.506404 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:57.507062 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:04:57.507085 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:04:57.508741 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:04:57.510603 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:04:57.510611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:04:57.510627 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:04:57.511065 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:04:57.511363 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:04:57.511369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:04:57.511374 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:04:57.511566 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:04:57.511981 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:04:57.511991 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:04:57.512352 (+ 361us) client_negotiation.cc:769] Sending connection context
0805 23:04:57.512583 (+ 231us) client_negotiation.cc:240] Negotiation successful
0805 23:04:57.512956 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":197,"threads_started":1}
I20260805 23:04:57.515523 3477 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41697
I20260805 23:04:57.515794 3477 heartbeater.cc:464] Registering TS with master...
I20260805 23:04:57.516409 3477 heartbeater.cc:511] Master 127.3.12.126:41697 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:04:57.518213 3208 ts_manager.cc:194] Registered new tserver with Master: dfa6cb9c779741799106c7d86f849637 (127.3.12.65:39917)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:04:57.519579 3208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:43281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:04:57.537571 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41697
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:04:57.654047 3483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:04:57.654366 3483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:04:57.654444 3483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:04:57.658623 3483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:04:57.658725 3483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:04:57.658816 3483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:04:57.664649 3483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:04:57.665792 3483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:04:57.666890 3483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:04:57.677281 3492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:57.677256 3491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:57.677297 3494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:04:57.677516 3483 server_base.cc:1034] running on GCE node
I20260805 23:04:57.678092 3483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:04:57.678776 3483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:04:57.679941 3483 hybrid_clock.cc:648] HybridClock initialized: now 1785971097679921 us; error 35 us; skew 500 ppm
Aug 05 23:04:57 dist-test-slave-2d19 krb5kdc[3131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:04:57.682863 3483 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:04:57.683717 3483 webserver.cc:460] Webserver started at http://127.3.12.66:44889/ using document root <none> and password file <none>
I20260805 23:04:57.684336 3483 fs_manager.cc:359] Metadata directory not provided
I20260805 23:04:57.684408 3483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:04:57.684685 3483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:04:57.686770 3483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "e4e5c4a2284f417dbc770b22572a21f4"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "1d7ec7c491e55ef3f4de5e3e0f9c044d"
server_key_iv: "9845d8421e7ee8f88f5ccb07052520d6"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.687309 3483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "e4e5c4a2284f417dbc770b22572a21f4"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "1d7ec7c491e55ef3f4de5e3e0f9c044d"
server_key_iv: "9845d8421e7ee8f88f5ccb07052520d6"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.691344 3483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260805 23:04:57.694041 3501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.695315 3483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 23:04:57.695461 3483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "e4e5c4a2284f417dbc770b22572a21f4"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "1d7ec7c491e55ef3f4de5e3e0f9c044d"
server_key_iv: "9845d8421e7ee8f88f5ccb07052520d6"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.695582 3483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:04:57.710919 3483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:04:57.713853 3483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:04:57.714062 3483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:04:57.714728 3483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:04:57.715580 3483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:04:57.715652 3483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.715724 3483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:04:57.715778 3483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.726169 3483 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:41111
I20260805 23:04:57.726197 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:41111 every 8 connection(s)
I20260805 23:04:57.727120 3483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:04:57.735461 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3483
I20260805 23:04:57.735605 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:04:57.735956 3121 external_mini_cluster.cc:1428] Setting key 3754edeebbcf74d9def4741425b62e67
Aug 05 23:04:57 dist-test-slave-2d19 krb5kdc[3131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:04:57.742177 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:57.729155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:57495 (local address 127.3.12.126:41697)
0805 23:04:57.729325 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0805 23:04:57.729329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:04:57.730226 (+ 897us) server_negotiation.cc:407] Connection header received
0805 23:04:57.731710 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:04:57.731714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:04:57.731781 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:04:57.731873 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:04:57.733176 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.733703 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.734863 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.735096 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.735215 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:57.738033 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:04:57.738038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:04:57.738041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:04:57.738071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:04:57.739782 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:57.740518 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:57.740521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:57.740522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:57.740601 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:57.741029 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:57.741031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:57.741032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:57.741201 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:04:57.741276 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0805 23:04:57.741920 (+ 644us) server_negotiation.cc:299] Negotiation successful
0805 23:04:57.742054 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 23:04:57.743377 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:57.729504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41697 (local address 127.3.12.66:57495)
0805 23:04:57.730054 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0805 23:04:57.730099 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:04:57.731447 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:04:57.732082 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:04:57.732089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:04:57.732534 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:04:57.733017 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:04:57.733031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.733837 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.733841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:04:57.734729 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:04:57.734738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:04:57.735260 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:04:57.735264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:04:57.735441 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:57.736190 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:04:57.736213 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:04:57.737849 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:04:57.739936 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:04:57.739945 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:04:57.739964 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:04:57.740401 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:04:57.740696 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:04:57.740701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:04:57.740704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:04:57.740896 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:04:57.741308 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:04:57.741317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:04:57.741659 (+ 342us) client_negotiation.cc:769] Sending connection context
0805 23:04:57.741864 (+ 205us) client_negotiation.cc:240] Negotiation successful
0805 23:04:57.742219 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":185,"threads_started":1}
I20260805 23:04:57.744720 3649 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41697
I20260805 23:04:57.744985 3649 heartbeater.cc:464] Registering TS with master...
I20260805 23:04:57.745479 3649 heartbeater.cc:511] Master 127.3.12.126:41697 requested a full tablet report, sending...
I20260805 23:04:57.746572 3208 ts_manager.cc:194] Registered new tserver with Master: e4e5c4a2284f417dbc770b22572a21f4 (127.3.12.66:41111)
I20260805 23:04:57.747155 3208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:57495
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:04:57.766647 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41697
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:37117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:04:57.919119 3657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:04:57.919423 3657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:04:57.919502 3657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:04:57.923810 3657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:04:57.923894 3657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:04:57.923987 3657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:04:57.930058 3657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:37117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:04:57.931193 3657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:04:57.932291 3657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:04:57.943688 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:57.943782 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:57.944216 3667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:04:57.944345 3657 server_base.cc:1034] running on GCE node
I20260805 23:04:57.944790 3657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:04:57.945504 3657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:04:57.946695 3657 hybrid_clock.cc:648] HybridClock initialized: now 1785971097946661 us; error 45 us; skew 500 ppm
Aug 05 23:04:57 dist-test-slave-2d19 krb5kdc[3131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:04:57.949781 3657 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:04:57.951068 3657 webserver.cc:460] Webserver started at http://127.3.12.67:42529/ using document root <none> and password file <none>
I20260805 23:04:57.951730 3657 fs_manager.cc:359] Metadata directory not provided
I20260805 23:04:57.951809 3657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:04:57.952136 3657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:04:57.954401 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "fa700c819d1f41f1b484d4bda72abbca"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "e7cf05d9f3ac40a08cea2500998809fb"
server_key_iv: "0e04b83bb251ac18c90e2266f8a6d206"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.955029 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "fa700c819d1f41f1b484d4bda72abbca"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "e7cf05d9f3ac40a08cea2500998809fb"
server_key_iv: "0e04b83bb251ac18c90e2266f8a6d206"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.959257 3657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260805 23:04:57.962203 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.963542 3657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 23:04:57.963671 3657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "fa700c819d1f41f1b484d4bda72abbca"
format_stamp: "Formatted at 2026-08-05 23:04:57 on dist-test-slave-2d19"
server_key: "e7cf05d9f3ac40a08cea2500998809fb"
server_key_iv: "0e04b83bb251ac18c90e2266f8a6d206"
server_key_version: "encryptionkey@0"
I20260805 23:04:57.963801 3657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:04:57.984184 3657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:04:57.987512 3657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:04:57.987808 3657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:04:57.988747 3657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:04:57.989777 3657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:04:57.989861 3657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:57.989938 3657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:04:57.989984 3657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:58.008880 3657 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:42819
I20260805 23:04:58.008996 3827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:42819 every 8 connection(s)
I20260805 23:04:58.009961 3657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:04:58.014881 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3657
I20260805 23:04:58.014986 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:04:58.015268 3121 external_mini_cluster.cc:1428] Setting key cde52ff3d9866a8aa6c00f2ab3a223d1
Aug 05 23:04:58 dist-test-slave-2d19 krb5kdc[3131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:04:58.026767 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:58.012129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:46607 (local address 127.3.12.126:41697)
0805 23:04:58.012305 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0805 23:04:58.012310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:04:58.013550 (+ 1240us) server_negotiation.cc:407] Connection header received
0805 23:04:58.015285 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:04:58.015289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:04:58.015353 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:04:58.015476 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:04:58.017364 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:58.017986 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:58.019423 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:58.019638 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:58.019722 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:58.022609 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:04:58.022613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:04:58.022615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:04:58.022639 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:04:58.024346 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:58.024896 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:58.024899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:58.024900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:58.024987 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:58.025578 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:58.025580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:58.025581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:58.025757 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:04:58.025834 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0805 23:04:58.026464 (+ 630us) server_negotiation.cc:299] Negotiation successful
0805 23:04:58.026624 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:04:58.027866 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:58.012512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41697 (local address 127.3.12.67:46607)
0805 23:04:58.013300 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0805 23:04:58.013348 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:04:58.015025 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:04:58.015672 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:04:58.015680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:04:58.016363 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:04:58.017199 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:04:58.017221 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:04:58.018151 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:04:58.018154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:04:58.019310 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:04:58.019321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:04:58.019802 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:04:58.019805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:04:58.020010 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:58.020747 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:04:58.020763 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:04:58.022439 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:04:58.024479 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:04:58.024484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:04:58.024497 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:04:58.024779 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:04:58.025143 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:04:58.025146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:04:58.025147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:04:58.025295 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:04:58.025854 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:04:58.025860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:04:58.026207 (+ 347us) client_negotiation.cc:769] Sending connection context
0805 23:04:58.026373 (+ 166us) client_negotiation.cc:240] Negotiation successful
0805 23:04:58.026789 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":226,"threads_started":1}
I20260805 23:04:58.029491 3829 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41697
I20260805 23:04:58.029795 3829 heartbeater.cc:464] Registering TS with master...
I20260805 23:04:58.030413 3829 heartbeater.cc:511] Master 127.3.12.126:41697 requested a full tablet report, sending...
I20260805 23:04:58.031484 3208 ts_manager.cc:194] Registered new tserver with Master: fa700c819d1f41f1b484d4bda72abbca (127.3.12.67:42819)
I20260805 23:04:58.032094 3208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:46607
I20260805 23:04:58.042239 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 23:04:58 dist-test-slave-2d19 krb5kdc[3131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 23:04:58 dist-test-slave-2d19 krb5kdc[3131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:04:58.073390 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:58.062131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.3.12.65:39917)
0805 23:04:58.062428 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0805 23:04:58.062438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 23:04:58.062589 (+ 151us) server_negotiation.cc:407] Connection header received
0805 23:04:58.062691 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:04:58.062696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:04:58.062898 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:04:58.063087 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:04:58.064049 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:58.065088 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:58.066098 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:58.066625 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:58.066769 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:58.069597 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:04:58.069607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:04:58.069622 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:04:58.069647 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:04:58.071583 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:58.071989 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:58.071994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:58.071995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:58.072079 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:58.072431 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:58.072435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:58.072436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:58.072707 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:04:58.072879 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0805 23:04:58.073066 (+ 187us) server_negotiation.cc:299] Negotiation successful
0805 23:04:58.073203 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
W20260805 23:04:58.074592 3357 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46840
Aug 05 23:04:58 dist-test-slave-2d19 krb5kdc[3131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 23:04:58 dist-test-slave-2d19 krb5kdc[3131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:04:58.104480 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:58.095461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.3.12.65:39917)
0805 23:04:58.095667 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0805 23:04:58.095672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:04:58.095697 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:04:58.095831 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:04:58.095834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:04:58.095879 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:04:58.095955 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:04:58.096914 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:58.097449 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:58.098326 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:58.098568 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:58.098685 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:58.101480 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:04:58.101485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:04:58.101487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:04:58.101512 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:04:58.102888 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:58.103332 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:58.103335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:58.103336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:58.103416 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:58.103662 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:58.103664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:58.103665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:58.103820 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:04:58.103982 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0805 23:04:58.104116 (+ 134us) server_negotiation.cc:299] Negotiation successful
0805 23:04:58.104214 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260805 23:04:58.105530 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3298
I20260805 23:04:58.119854 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3483
I20260805 23:04:58.130594 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3657
I20260805 23:04:58.149515 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3152
2026-08-05T23:04:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2664 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:04:58 dist-test-slave-2d19 krb5kdc[3847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:04:58 dist-test-slave-2d19 krb5kdc[3847](info): set up 2 sockets
Aug 05 23:04:58 dist-test-slave-2d19 krb5kdc[3847](info): commencing operation
krb5kdc: starting...
W20260805 23:04:59.404711 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:04:59 dist-test-slave-2d19 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:04:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:04:59.539453 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:45711
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:45711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:04:59.653343 3869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:04:59.653635 3869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:04:59.653676 3869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:04:59.657609 3869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:04:59.657665 3869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:04:59.657684 3869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:04:59.657701 3869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:04:59.657716 3869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:04:59.662967 3869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:45711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:45711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:04:59.663970 3869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:04:59.664994 3869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:04:59.673633 3879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:59.673653 3875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:59.673653 3876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:04:59.673720 3869 server_base.cc:1034] running on GCE node
I20260805 23:04:59.674436 3869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:04:59.675628 3869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:04:59.676855 3869 hybrid_clock.cc:648] HybridClock initialized: now 1785971099676841 us; error 25 us; skew 500 ppm
Aug 05 23:04:59 dist-test-slave-2d19 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971099, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:04:59.679688 3869 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:04:59.680608 3869 webserver.cc:460] Webserver started at http://127.3.12.126:35545/ using document root <none> and password file <none>
I20260805 23:04:59.681145 3869 fs_manager.cc:359] Metadata directory not provided
I20260805 23:04:59.681205 3869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:04:59.681370 3869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:04:59.683377 3869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "117f1520099b4184b97f50e15fb2d14e"
format_stamp: "Formatted at 2026-08-05 23:04:59 on dist-test-slave-2d19"
server_key: "388076c2ba8e790a2a346b02d4bbcab5"
server_key_iv: "d95d680c7f57e7f1b2c3197339fe7839"
server_key_version: "encryptionkey@0"
I20260805 23:04:59.683910 3869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "117f1520099b4184b97f50e15fb2d14e"
format_stamp: "Formatted at 2026-08-05 23:04:59 on dist-test-slave-2d19"
server_key: "388076c2ba8e790a2a346b02d4bbcab5"
server_key_iv: "d95d680c7f57e7f1b2c3197339fe7839"
server_key_version: "encryptionkey@0"
I20260805 23:04:59.687747 3869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:04:59.690328 3887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:04:59.691613 3869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:04:59.691774 3869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "117f1520099b4184b97f50e15fb2d14e"
format_stamp: "Formatted at 2026-08-05 23:04:59 on dist-test-slave-2d19"
server_key: "388076c2ba8e790a2a346b02d4bbcab5"
server_key_iv: "d95d680c7f57e7f1b2c3197339fe7839"
server_key_version: "encryptionkey@0"
I20260805 23:04:59.691886 3869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:04:59.728084 3869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:04:59.731675 3869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:04:59.731925 3869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:04:59.739701 3869 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:45711
I20260805 23:04:59.739717 3963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:45711 every 8 connection(s)
I20260805 23:04:59.740789 3869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:04:59.743646 3964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:04:59.747130 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3869
I20260805 23:04:59.747248 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:04:59.747511 3121 external_mini_cluster.cc:1428] Setting key 12aa5ce890a45320001e4128fe91e09f
I20260805 23:04:59.749775 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e: Bootstrap starting.
I20260805 23:04:59.752079 3964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e: Neither blocks nor log segments found. Creating new log.
I20260805 23:04:59.752861 3964 log.cc:824] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e: Log is configured to *not* fsync() on all Append() calls
I20260805 23:04:59.754761 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e: No bootstrap required, opened a new log
Aug 05 23:04:59 dist-test-slave-2d19 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:04:59.758553 3964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f1520099b4184b97f50e15fb2d14e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45711 } }
I20260805 23:04:59.758816 3964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:04:59.758889 3964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117f1520099b4184b97f50e15fb2d14e, State: Initialized, Role: FOLLOWER
I20260805 23:04:59.759394 3964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f1520099b4184b97f50e15fb2d14e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45711 } }
I20260805 23:04:59.759523 3964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:04:59.759608 3964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:04:59.759711 3964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:04:59.760684 3964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f1520099b4184b97f50e15fb2d14e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45711 } }
I20260805 23:04:59.761027 3964 leader_election.cc:302] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 117f1520099b4184b97f50e15fb2d14e; no voters:
I20260805 23:04:59.761355 3964 leader_election.cc:288] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:04:59.761488 3972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:04:59.761736 3972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [term 1 LEADER]: Becoming Leader. State: Replica: 117f1520099b4184b97f50e15fb2d14e, State: Running, Role: LEADER
I20260805 23:04:59.762137 3972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f1520099b4184b97f50e15fb2d14e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45711 } }
I20260805 23:04:59.762733 3964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:04:59.763528 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:04:59.749095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.3.12.126:45711)
0805 23:04:59.749548 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0805 23:04:59.749564 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:04:59.749603 (+ 39us) server_negotiation.cc:407] Connection header received
0805 23:04:59.750321 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:04:59.750340 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:04:59.750696 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:04:59.751044 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:04:59.751910 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:59.753102 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:59.753991 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:04:59.754498 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:04:59.754695 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:04:59.757881 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:04:59.757886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:04:59.757902 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:04:59.757930 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:04:59.759936 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:59.760481 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:59.760487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:59.760494 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:59.760617 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:04:59.760930 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:04:59.760933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:04:59.760935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:04:59.761484 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:04:59.761712 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0805 23:04:59.762013 (+ 301us) server_negotiation.cc:299] Negotiation successful
0805 23:04:59.762331 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":136,"threads_started":1}
I20260805 23:04:59.763588 3972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 117f1520099b4184b97f50e15fb2d14e. Latest consensus state: current_term: 1 leader_uuid: "117f1520099b4184b97f50e15fb2d14e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f1520099b4184b97f50e15fb2d14e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45711 } } }
I20260805 23:04:59.763695 3972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [sys.catalog]: This master's current role is: LEADER
I20260805 23:04:59.765249 3973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "117f1520099b4184b97f50e15fb2d14e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "117f1520099b4184b97f50e15fb2d14e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45711 } } }
I20260805 23:04:59.765867 3973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e [sys.catalog]: This master's current role is: LEADER
W20260805 23:04:59.768863 3991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:04:59.768975 3991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:04:59.769064 3984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:04:59.769814 3984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:04:59.775681 3984 catalog_manager.cc:1347] Generated new cluster ID: 8cb7886106554df68a30edffa6caa771
I20260805 23:04:59.775763 3984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:04:59.797537 3984 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:04:59.798573 3984 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:04:59.803838 3984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 117f1520099b4184b97f50e15fb2d14e: Generated new TSK 0
I20260805 23:04:59.804590 3984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:04:59.848747 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45711
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:04:59.965096 3998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:04:59.965397 3998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:04:59.965478 3998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:04:59.969637 3998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:04:59.969727 3998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:04:59.969820 3998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:04:59.975243 3998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:04:59.976399 3998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:04:59.977454 3998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:04:59.987273 4004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:59.987263 4003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:04:59.987263 4009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:04:59.987648 3998 server_base.cc:1034] running on GCE node
I20260805 23:04:59.988097 3998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:04:59.988823 3998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:04:59.989995 3998 hybrid_clock.cc:648] HybridClock initialized: now 1785971099989977 us; error 35 us; skew 500 ppm
Aug 05 23:04:59 dist-test-slave-2d19 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971099, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:04:59.992964 3998 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:04:59.993876 3998 webserver.cc:460] Webserver started at http://127.3.12.65:36095/ using document root <none> and password file <none>
I20260805 23:04:59.994475 3998 fs_manager.cc:359] Metadata directory not provided
I20260805 23:04:59.994575 3998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:04:59.994828 3998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:04:59.997154 3998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "3445bd31ad07492eb9366bf33d09002d"
format_stamp: "Formatted at 2026-08-05 23:04:59 on dist-test-slave-2d19"
server_key: "6ba8193a9c7d26e8612db7df25081616"
server_key_iv: "4ae174ac1835ccb9d8f628377114a727"
server_key_version: "encryptionkey@0"
I20260805 23:04:59.997767 3998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "3445bd31ad07492eb9366bf33d09002d"
format_stamp: "Formatted at 2026-08-05 23:04:59 on dist-test-slave-2d19"
server_key: "6ba8193a9c7d26e8612db7df25081616"
server_key_iv: "4ae174ac1835ccb9d8f628377114a727"
server_key_version: "encryptionkey@0"
I20260805 23:05:00.001746 3998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:00.004387 4021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.005733 3998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:00.005888 3998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "3445bd31ad07492eb9366bf33d09002d"
format_stamp: "Formatted at 2026-08-05 23:04:59 on dist-test-slave-2d19"
server_key: "6ba8193a9c7d26e8612db7df25081616"
server_key_iv: "4ae174ac1835ccb9d8f628377114a727"
server_key_version: "encryptionkey@0"
I20260805 23:05:00.006009 3998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:00.024298 3998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:00.027616 3998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:00.027848 3998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:00.028594 3998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:00.029510 3998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:00.029587 3998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.029660 3998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:00.029711 3998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.039477 3998 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:45575
I20260805 23:05:00.039498 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:45575 every 8 connection(s)
I20260805 23:05:00.040462 3998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:00.046602 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3998
I20260805 23:05:00.046738 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:00.047120 3121 external_mini_cluster.cc:1428] Setting key 41823310b6570cc24b079df50f223c3c
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971099, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:00.055323 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.042310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:34919 (local address 127.3.12.126:45711)
0805 23:05:00.042456 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.042460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.043261 (+ 801us) server_negotiation.cc:407] Connection header received
0805 23:05:00.044741 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.044745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.044809 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.044896 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:00.046098 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.046693 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.048222 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.048547 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.048658 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.051231 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:00.051235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:00.051237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:00.051260 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:00.052861 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.053579 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.053582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.053583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.053678 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.054136 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.054139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.054141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.054323 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:00.054403 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.055084 (+ 681us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.055197 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:05:00.056643 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.042641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:45711 (local address 127.3.12.65:34919)
0805 23:05:00.043087 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:00.043133 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:00.044513 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:00.045086 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:00.045094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:00.045457 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:00.045937 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.045950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.046893 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.046898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.048094 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.048105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.048719 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.048724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.048895 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.049542 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:00.049559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:00.051052 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:00.052991 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:00.053001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:00.053028 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:00.053466 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:00.053807 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:00.053812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:00.053815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:00.054029 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:00.054438 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:00.054448 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:00.054799 (+ 351us) client_negotiation.cc:769] Sending connection context
0805 23:05:00.055052 (+ 253us) client_negotiation.cc:240] Negotiation successful
0805 23:05:00.055449 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260805 23:05:00.058028 4181 heartbeater.cc:347] Connected to a master server at 127.3.12.126:45711
I20260805 23:05:00.058281 4181 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:00.058836 4181 heartbeater.cc:511] Master 127.3.12.126:45711 requested a full tablet report, sending...
I20260805 23:05:00.060478 3912 ts_manager.cc:194] Registered new tserver with Master: 3445bd31ad07492eb9366bf33d09002d (127.3.12.65:45575)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:05:00.062028 3912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:34919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:00.079663 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45711
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:00.196290 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:00.196653 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:00.196745 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:00.200876 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:00.200968 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:00.201063 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:00.206421 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:00.207655 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:00.208827 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:00.219398 4194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:00.219447 4196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:00.219475 4186 server_base.cc:1034] running on GCE node
W20260805 23:05:00.219738 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:00.220206 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:00.220903 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:00.222079 4186 hybrid_clock.cc:648] HybridClock initialized: now 1785971100222065 us; error 31 us; skew 500 ppm
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:00.225113 4186 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:00.226074 4186 webserver.cc:460] Webserver started at http://127.3.12.66:39031/ using document root <none> and password file <none>
I20260805 23:05:00.226677 4186 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:00.226773 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:00.227015 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:00.229267 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "9c4e8f5e81974a6293473b9cfa54ddfe"
format_stamp: "Formatted at 2026-08-05 23:05:00 on dist-test-slave-2d19"
server_key: "d1366604f1ff829ab3a9b08bbe82862a"
server_key_iv: "64e547e43a85e0e2e39e19bf3c45beaa"
server_key_version: "encryptionkey@0"
I20260805 23:05:00.229950 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "9c4e8f5e81974a6293473b9cfa54ddfe"
format_stamp: "Formatted at 2026-08-05 23:05:00 on dist-test-slave-2d19"
server_key: "d1366604f1ff829ab3a9b08bbe82862a"
server_key_iv: "64e547e43a85e0e2e39e19bf3c45beaa"
server_key_version: "encryptionkey@0"
I20260805 23:05:00.234030 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:00.236629 4210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.237859 4186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:05:00.238013 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "9c4e8f5e81974a6293473b9cfa54ddfe"
format_stamp: "Formatted at 2026-08-05 23:05:00 on dist-test-slave-2d19"
server_key: "d1366604f1ff829ab3a9b08bbe82862a"
server_key_iv: "64e547e43a85e0e2e39e19bf3c45beaa"
server_key_version: "encryptionkey@0"
I20260805 23:05:00.238133 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:00.264992 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:00.268537 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:00.268714 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:00.269248 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:00.270051 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:00.270099 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.270140 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:00.270159 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.279202 4186 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:43843
I20260805 23:05:00.279268 4367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:43843 every 8 connection(s)
I20260805 23:05:00.280206 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:00.287221 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 4186
I20260805 23:05:00.287325 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:00.287624 3121 external_mini_cluster.cc:1428] Setting key fb1c4c2edbd5a8b099839aa194a8ac00
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:00.295840 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.282207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:44393 (local address 127.3.12.126:45711)
0805 23:05:00.282348 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.282351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.283268 (+ 917us) server_negotiation.cc:407] Connection header received
0805 23:05:00.284767 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.284770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.284832 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.284948 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:00.286164 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.286709 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.288076 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.288450 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.288560 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.291579 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:00.291582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:00.291584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:00.291608 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:00.293232 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.293935 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.293938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.293939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.294015 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.294450 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.294452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.294453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.294627 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:00.294719 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.295574 (+ 855us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.295712 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:05:00.297345 4370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.282639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:45711 (local address 127.3.12.66:44393)
0805 23:05:00.283107 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:00.283159 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:00.284538 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:00.285134 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:00.285148 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:00.285530 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:00.286007 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.286021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.286853 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.286856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.287918 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.287929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.288624 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.288628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.288842 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.289523 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:00.289543 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:00.291384 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:00.293366 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:00.293377 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:00.293397 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:00.293828 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:00.294115 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:00.294122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:00.294127 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:00.294343 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:00.294767 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:00.294779 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:00.295246 (+ 467us) client_negotiation.cc:769] Sending connection context
0805 23:05:00.295511 (+ 265us) client_negotiation.cc:240] Negotiation successful
0805 23:05:00.295944 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20260805 23:05:00.298763 4368 heartbeater.cc:347] Connected to a master server at 127.3.12.126:45711
I20260805 23:05:00.299047 4368 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:00.299606 4368 heartbeater.cc:511] Master 127.3.12.126:45711 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:05:00.300820 3912 ts_manager.cc:194] Registered new tserver with Master: 9c4e8f5e81974a6293473b9cfa54ddfe (127.3.12.66:43843)
I20260805 23:05:00.301515 3912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:44393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:00.321136 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45711
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:00.433761 4373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:00.434019 4373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:00.434062 4373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:00.437913 4373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:00.437973 4373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:00.438030 4373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:00.443249 4373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:00.444284 4373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:00.445312 4373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:00.455049 4382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:00.455049 4380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:00.455113 4384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:00.455322 4373 server_base.cc:1034] running on GCE node
I20260805 23:05:00.455752 4373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:00.456468 4373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:00.457667 4373 hybrid_clock.cc:648] HybridClock initialized: now 1785971100457654 us; error 29 us; skew 500 ppm
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:00.460649 4373 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:00.461591 4373 webserver.cc:460] Webserver started at http://127.3.12.67:42937/ using document root <none> and password file <none>
I20260805 23:05:00.462194 4373 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:00.462296 4373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:00.462558 4373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:00.464866 4373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "f80205c475e74953af8ad3f26b4459d2"
format_stamp: "Formatted at 2026-08-05 23:05:00 on dist-test-slave-2d19"
server_key: "13319231cd8f7f14687ab0377a8976b7"
server_key_iv: "a6bc465ee49c394ba1f2bdd6fefe370b"
server_key_version: "encryptionkey@0"
I20260805 23:05:00.465441 4373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "f80205c475e74953af8ad3f26b4459d2"
format_stamp: "Formatted at 2026-08-05 23:05:00 on dist-test-slave-2d19"
server_key: "13319231cd8f7f14687ab0377a8976b7"
server_key_iv: "a6bc465ee49c394ba1f2bdd6fefe370b"
server_key_version: "encryptionkey@0"
I20260805 23:05:00.469434 4373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260805 23:05:00.472040 4393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.473519 4373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:05:00.473681 4373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "f80205c475e74953af8ad3f26b4459d2"
format_stamp: "Formatted at 2026-08-05 23:05:00 on dist-test-slave-2d19"
server_key: "13319231cd8f7f14687ab0377a8976b7"
server_key_iv: "a6bc465ee49c394ba1f2bdd6fefe370b"
server_key_version: "encryptionkey@0"
I20260805 23:05:00.473820 4373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:00.491993 4373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:00.495807 4373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:00.496032 4373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:00.496790 4373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:00.497769 4373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:00.497840 4373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.497907 4373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:00.497959 4373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:00.507099 4373 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:36337
I20260805 23:05:00.507124 4557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:36337 every 8 connection(s)
I20260805 23:05:00.508143 4373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:00.518184 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 4373
I20260805 23:05:00.518352 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971100, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:00.518663 3121 external_mini_cluster.cc:1428] Setting key 391bb81be7a5553e42509a1d50a35c9d
I20260805 23:05:00.523268 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.510125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:36413 (local address 127.3.12.126:45711)
0805 23:05:00.510290 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.510295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.511227 (+ 932us) server_negotiation.cc:407] Connection header received
0805 23:05:00.512654 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.512657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.512706 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.512786 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:00.514005 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.514558 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.515707 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.515930 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.516033 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.519315 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:00.519319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:00.519321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:00.519349 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:00.520993 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.521590 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.521594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.521596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.521690 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.522075 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.522079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.522081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.522294 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:00.522406 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.522959 (+ 553us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.523094 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:05:00.524142 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.510600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:45711 (local address 127.3.12.67:36413)
0805 23:05:00.511067 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:00.511110 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:00.512433 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:00.512965 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:00.512972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:00.513339 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:00.513835 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.513849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.514687 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.514690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.515572 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.515581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.516374 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.516383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.516562 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.517223 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:00.517240 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:00.519115 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:00.521135 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:00.521141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:00.521159 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:00.521473 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:00.521790 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:00.521793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:00.521795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:00.521971 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:00.522423 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:00.522431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:00.522712 (+ 281us) client_negotiation.cc:769] Sending connection context
0805 23:05:00.522916 (+ 204us) client_negotiation.cc:240] Negotiation successful
0805 23:05:00.523276 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":151,"threads_started":1}
I20260805 23:05:00.525367 4558 heartbeater.cc:347] Connected to a master server at 127.3.12.126:45711
I20260805 23:05:00.525599 4558 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:00.526068 4558 heartbeater.cc:511] Master 127.3.12.126:45711 requested a full tablet report, sending...
I20260805 23:05:00.527081 3912 ts_manager.cc:194] Registered new tserver with Master: f80205c475e74953af8ad3f26b4459d2 (127.3.12.67:36337)
I20260805 23:05:00.527662 3912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:36413
I20260805 23:05:00.532836 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:05:00.543922 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.536332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.3.12.126:45711)
0805 23:05:00.536483 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.536489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.536676 (+ 187us) server_negotiation.cc:407] Connection header received
0805 23:05:00.536758 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.536760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.536802 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.536869 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:00.537542 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.538057 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.538870 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.539071 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.539147 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.540594 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:00.540597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:00.540599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:00.540622 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:00.542039 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.542431 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.542434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.542436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.542565 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.542828 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.542830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.542831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.543174 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:00.543347 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.543546 (+ 199us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.543713 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:05:00.549094 3912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:00.552134 3912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:00.567781 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.561496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:36337 (local address 127.0.0.1:41088)
0805 23:05:00.562456 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:00.562470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:00.562568 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:00.563444 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:00.563450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:00.563474 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:00.563582 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.563589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.565126 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.565131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.566250 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.566260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.567131 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.567135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.567257 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.567356 (+ 99us) client_negotiation.cc:769] Sending connection context
0805 23:05:00.567475 (+ 119us) client_negotiation.cc:240] Negotiation successful
0805 23:05:00.567625 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":90,"thread_start_us":96,"threads_started":1}
I20260805 23:05:00.567804 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.562456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.3.12.67:36337)
0805 23:05:00.562967 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.562980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.563005 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:05:00.563088 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.563096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.563304 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.563442 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:00.563708 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.564905 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.566416 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.566999 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.567089 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.567253 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.567531 (+ 278us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.567642 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":150,"threads_started":1}
I20260805 23:05:00.567836 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.562020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.3.12.66:43843)
0805 23:05:00.562423 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.562434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.562457 (+ 23us) server_negotiation.cc:407] Connection header received
0805 23:05:00.562537 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.562543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.562760 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.563581 (+ 821us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:00.563763 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.564638 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.566345 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.567118 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.567235 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.567389 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.567594 (+ 205us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.567690 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":163,"threads_started":1}
I20260805 23:05:00.568383 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.561124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:43843 (local address 127.0.0.1:43882)
0805 23:05:00.561587 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:00.561635 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:00.561949 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:00.562943 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:00.562950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:00.562994 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:00.563230 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.563239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.564772 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.564778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.566194 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.566207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.567252 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.567256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.567397 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.567416 (+ 19us) client_negotiation.cc:769] Sending connection context
0805 23:05:00.567512 (+ 96us) client_negotiation.cc:240] Negotiation successful
0805 23:05:00.568191 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":664,"thread_start_us":105,"threads_started":1}
I20260805 23:05:00.568969 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.561224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:45575 (local address 127.0.0.1:42510)
0805 23:05:00.562012 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:00.562029 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:00.562116 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:00.562943 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:00.562950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:00.563032 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:00.563377 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.563405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.565063 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.565067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.566218 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.566225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.568569 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.568573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.568656 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.568680 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 23:05:00.568748 (+ 68us) client_negotiation.cc:240] Negotiation successful
0805 23:05:00.568829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":211,"spinlock_wait_cycles":2432,"thread_start_us":135,"threads_started":1}
I20260805 23:05:00.569836 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.562119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.3.12.65:45575)
0805 23:05:00.562423 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.562432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.562457 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:05:00.562537 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.562543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.562760 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.563205 (+ 445us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:00.563529 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.564905 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.567472 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.568412 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.569315 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.569480 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.569536 (+ 56us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.569628 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260805 23:05:00.572125 4089 tablet_service.cc:1467] Processing CreateTablet for tablet 1a46fe8c59ac436fb930ae1ae2e4c816 (DEFAULT_TABLE table=test-table [id=720f8e20b3fc4a5182eda33a330c9f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:00.572022 4273 tablet_service.cc:1467] Processing CreateTablet for tablet 1a46fe8c59ac436fb930ae1ae2e4c816 (DEFAULT_TABLE table=test-table [id=720f8e20b3fc4a5182eda33a330c9f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:00.572695 4465 tablet_service.cc:1467] Processing CreateTablet for tablet 1a46fe8c59ac436fb930ae1ae2e4c816 (DEFAULT_TABLE table=test-table [id=720f8e20b3fc4a5182eda33a330c9f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:00.573299 4273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a46fe8c59ac436fb930ae1ae2e4c816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:00.573299 4089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a46fe8c59ac436fb930ae1ae2e4c816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:00.573712 4465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a46fe8c59ac436fb930ae1ae2e4c816. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:00.581025 4582 tablet_bootstrap.cc:492] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d: Bootstrap starting.
I20260805 23:05:00.582928 4584 tablet_bootstrap.cc:492] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe: Bootstrap starting.
I20260805 23:05:00.582928 4585 tablet_bootstrap.cc:492] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2: Bootstrap starting.
I20260805 23:05:00.583837 4582 tablet_bootstrap.cc:654] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:00.585181 4582 log.cc:824] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:00.585908 4585 tablet_bootstrap.cc:654] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:00.586529 4584 tablet_bootstrap.cc:654] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:00.587311 4582 tablet_bootstrap.cc:492] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d: No bootstrap required, opened a new log
I20260805 23:05:00.587538 4582 ts_tablet_manager.cc:1397] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260805 23:05:00.587834 4585 log.cc:824] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:00.587833 4584 log.cc:824] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:00.590075 4584 tablet_bootstrap.cc:492] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe: No bootstrap required, opened a new log
I20260805 23:05:00.590101 4585 tablet_bootstrap.cc:492] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2: No bootstrap required, opened a new log
I20260805 23:05:00.590296 4584 ts_tablet_manager.cc:1397] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260805 23:05:00.590296 4585 ts_tablet_manager.cc:1397] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260805 23:05:00.592783 4582 raft_consensus.cc:348] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.593044 4582 raft_consensus.cc:374] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:00.593120 4582 raft_consensus.cc:729] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3445bd31ad07492eb9366bf33d09002d, State: Initialized, Role: FOLLOWER
I20260805 23:05:00.593627 4582 consensus_queue.cc:260] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.594425 4181 heartbeater.cc:503] Master 127.3.12.126:45711 was elected leader, sending a full tablet report...
I20260805 23:05:00.594801 4582 ts_tablet_manager.cc:1428] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260805 23:05:00.595701 4584 raft_consensus.cc:348] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.595860 4585 raft_consensus.cc:348] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.596135 4584 raft_consensus.cc:374] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:00.596174 4585 raft_consensus.cc:374] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:00.596195 4584 raft_consensus.cc:729] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c4e8f5e81974a6293473b9cfa54ddfe, State: Initialized, Role: FOLLOWER
I20260805 23:05:00.596282 4585 raft_consensus.cc:729] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80205c475e74953af8ad3f26b4459d2, State: Initialized, Role: FOLLOWER
I20260805 23:05:00.596717 4585 consensus_queue.cc:260] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.596717 4584 consensus_queue.cc:260] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.597790 4584 ts_tablet_manager.cc:1428] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260805 23:05:00.597790 4585 ts_tablet_manager.cc:1428] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260805 23:05:00.597786 4558 heartbeater.cc:503] Master 127.3.12.126:45711 was elected leader, sending a full tablet report...
I20260805 23:05:00.598366 4368 heartbeater.cc:503] Master 127.3.12.126:45711 was elected leader, sending a full tablet report...
I20260805 23:05:00.707430 4591 raft_consensus.cc:482] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:00.707804 4591 raft_consensus.cc:504] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.709257 4591 leader_election.cc:288] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c4e8f5e81974a6293473b9cfa54ddfe (127.3.12.66:43843), 3445bd31ad07492eb9366bf33d09002d (127.3.12.65:45575)
I20260805 23:05:00.713641 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.709724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:45303 (local address 127.3.12.65:45575)
0805 23:05:00.709870 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.709874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.709890 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:05:00.710020 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.710022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.710067 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.710192 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:00.710551 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.711153 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.712246 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.713027 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.713136 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.713211 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.713447 (+ 236us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.713529 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:05:00.713642 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.709537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:45575 (local address 127.3.12.67:45303)
0805 23:05:00.709771 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:00.709788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:00.709892 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:00.710207 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:00.710224 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:00.710242 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:00.710420 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.710427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.711271 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.711274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.712101 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.712111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.713153 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.713156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.713287 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.713316 (+ 29us) client_negotiation.cc:769] Sending connection context
0805 23:05:00.713426 (+ 110us) client_negotiation.cc:240] Negotiation successful
0805 23:05:00.713500 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":24}
I20260805 23:05:00.714121 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.709697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:51103 (local address 127.3.12.66:43843)
0805 23:05:00.709840 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.709844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.710201 (+ 357us) server_negotiation.cc:407] Connection header received
0805 23:05:00.710361 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.710370 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.710414 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.710504 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:00.710886 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.711373 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.712820 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.713475 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.713597 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.713664 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.713943 (+ 279us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.714015 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260805 23:05:00.714186 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.709569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:43843 (local address 127.3.12.67:51103)
0805 23:05:00.710059 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:00.710070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:00.710241 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:00.710523 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:00.710525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:00.710540 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:00.710762 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.710769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.711497 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.711501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.712676 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:00.712687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.713603 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.713607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:00.713788 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.713821 (+ 33us) client_negotiation.cc:769] Sending connection context
0805 23:05:00.713923 (+ 102us) client_negotiation.cc:240] Negotiation successful
0805 23:05:00.713993 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":1,"thread_start_us":156,"threads_started":1}
I20260805 23:05:00.714387 4115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a46fe8c59ac436fb930ae1ae2e4c816" candidate_uuid: "f80205c475e74953af8ad3f26b4459d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3445bd31ad07492eb9366bf33d09002d" is_pre_election: true
I20260805 23:05:00.714740 4115 raft_consensus.cc:2454] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f80205c475e74953af8ad3f26b4459d2 in term 0.
I20260805 23:05:00.714926 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a46fe8c59ac436fb930ae1ae2e4c816" candidate_uuid: "f80205c475e74953af8ad3f26b4459d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" is_pre_election: true
I20260805 23:05:00.715294 4304 raft_consensus.cc:2454] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f80205c475e74953af8ad3f26b4459d2 in term 0.
I20260805 23:05:00.715344 4397 leader_election.cc:302] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3445bd31ad07492eb9366bf33d09002d, f80205c475e74953af8ad3f26b4459d2; no voters:
I20260805 23:05:00.715643 4591 raft_consensus.cc:2781] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:00.715729 4591 raft_consensus.cc:482] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:00.715795 4591 raft_consensus.cc:3037] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:00.717209 4591 raft_consensus.cc:504] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.717589 4591 leader_election.cc:288] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [CANDIDATE]: Term 1 election: Requested vote from peers 9c4e8f5e81974a6293473b9cfa54ddfe (127.3.12.66:43843), 3445bd31ad07492eb9366bf33d09002d (127.3.12.65:45575)
I20260805 23:05:00.717953 4304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a46fe8c59ac436fb930ae1ae2e4c816" candidate_uuid: "f80205c475e74953af8ad3f26b4459d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe"
I20260805 23:05:00.718044 4115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a46fe8c59ac436fb930ae1ae2e4c816" candidate_uuid: "f80205c475e74953af8ad3f26b4459d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3445bd31ad07492eb9366bf33d09002d"
I20260805 23:05:00.718093 4304 raft_consensus.cc:3037] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:00.718211 4115 raft_consensus.cc:3037] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:00.719455 4115 raft_consensus.cc:2454] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f80205c475e74953af8ad3f26b4459d2 in term 1.
I20260805 23:05:00.719492 4304 raft_consensus.cc:2454] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f80205c475e74953af8ad3f26b4459d2 in term 1.
I20260805 23:05:00.719758 4397 leader_election.cc:302] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3445bd31ad07492eb9366bf33d09002d, f80205c475e74953af8ad3f26b4459d2; no voters:
I20260805 23:05:00.719928 4591 raft_consensus.cc:2781] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:00.720212 4591 raft_consensus.cc:686] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [term 1 LEADER]: Becoming Leader. State: Replica: f80205c475e74953af8ad3f26b4459d2, State: Running, Role: LEADER
I20260805 23:05:00.720616 4591 consensus_queue.cc:237] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } }
I20260805 23:05:00.723907 3914 catalog_manager.cc:5495] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f80205c475e74953af8ad3f26b4459d2 (127.3.12.67). New cstate: current_term: 1 leader_uuid: "f80205c475e74953af8ad3f26b4459d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c4e8f5e81974a6293473b9cfa54ddfe" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3445bd31ad07492eb9366bf33d09002d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f80205c475e74953af8ad3f26b4459d2" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 36337 } health_report { overall_health: HEALTHY } } }
W20260805 23:05:00.759161 4559 tablet.cc:2369] T 1a46fe8c59ac436fb930ae1ae2e4c816 P f80205c475e74953af8ad3f26b4459d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:05:00.767773 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.751662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.3.12.65:45575)
0805 23:05:00.751832 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.751837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.752093 (+ 256us) server_negotiation.cc:407] Connection header received
0805 23:05:00.752221 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.752223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.752314 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.752395 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:00.753334 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.753843 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.754775 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.755045 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.755139 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.763654 (+ 8515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:00.763693 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:00.763713 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:00.763755 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:00.765888 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.766333 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.766339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.766340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.766421 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.766708 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.766710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.766711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.766967 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:00.767120 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.767405 (+ 285us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.767574 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260805 23:05:00.781463 4369 tablet.cc:2369] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 9c4e8f5e81974a6293473b9cfa54ddfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260805 23:05:00.791850 4182 tablet.cc:2369] T 1a46fe8c59ac436fb930ae1ae2e4c816 P 3445bd31ad07492eb9366bf33d09002d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:05:00.798097 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.789142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.3.12.65:45575)
0805 23:05:00.789279 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.789284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.789465 (+ 181us) server_negotiation.cc:407] Connection header received
0805 23:05:00.789651 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.789654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.789700 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.789784 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:00.790478 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.790992 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.791918 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.792225 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.792397 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.794830 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:00.794833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:00.794835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:00.794859 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:00.796290 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.796756 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.796763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.796768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.796880 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.797181 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.797186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.797188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.797364 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:00.797513 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.797802 (+ 289us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.797929 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[3847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:05:00.826134 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:00.817714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.3.12.65:45575)
0805 23:05:00.817870 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:00.817875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:00.818040 (+ 165us) server_negotiation.cc:407] Connection header received
0805 23:05:00.818202 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:00.818204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:00.818252 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:00.818329 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:00.819017 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.819505 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.820423 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:00.820651 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:00.820737 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:00.823395 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:00.823398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:00.823400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:00.823424 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:00.824734 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.825123 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.825126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.825127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.825199 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:00.825442 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:00.825444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:00.825445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:00.825604 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:00.825701 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:00.825906 (+ 205us) server_negotiation.cc:299] Negotiation successful
0805 23:05:00.826011 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260805 23:05:00.827659 4063 tablet_service.cc:2908] Rejecting scan request for tablet 1a46fe8c59ac436fb930ae1ae2e4c816: Uninitialized: clean time has not yet been initialized
I20260805 23:05:00.829833 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3998
I20260805 23:05:00.845885 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 4186
I20260805 23:05:00.861420 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 4373
I20260805 23:05:00.875129 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3869
2026-08-05T23:05:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2729 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[4616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[4616](info): set up 2 sockets
Aug 05 23:05:00 dist-test-slave-2d19 krb5kdc[4616](info): commencing operation
krb5kdc: starting...
W20260805 23:05:02.115446 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:02 dist-test-slave-2d19 krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:02.256155 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41011
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:41011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:02.375156 4633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:02.375442 4633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:02.375483 4633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:02.379491 4633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:02.379559 4633 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260805 23:05:02.379577 4633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:02.379599 4633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:02.379616 4633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:02.379631 4633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:02.385098 4633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:41011
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:02.386233 4633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:02.387252 4633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:02.396413 4644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:02.396435 4643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:02.396435 4646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:02.396495 4633 server_base.cc:1034] running on GCE node
I20260805 23:05:02.397418 4633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:02.398535 4633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:02.399715 4633 hybrid_clock.cc:648] HybridClock initialized: now 1785971102399695 us; error 34 us; skew 500 ppm
Aug 05 23:05:02 dist-test-slave-2d19 krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971102, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:02.402747 4633 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:02.403674 4633 webserver.cc:460] Webserver started at http://127.3.12.126:37503/ using document root <none> and password file <none>
I20260805 23:05:02.404228 4633 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:02.404348 4633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:02.404543 4633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:02.406641 4633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "5fa746eed6584d6f8b4f85dac7771686"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "660a880d7f52010aa86498bd7208619b"
server_key_iv: "c6b9162f85ff3ffe7ed711fd9b014bdb"
server_key_version: "encryptionkey@0"
I20260805 23:05:02.407178 4633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "5fa746eed6584d6f8b4f85dac7771686"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "660a880d7f52010aa86498bd7208619b"
server_key_iv: "c6b9162f85ff3ffe7ed711fd9b014bdb"
server_key_version: "encryptionkey@0"
I20260805 23:05:02.411096 4633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:05:02.413806 4653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:02.415122 4633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:02.415232 4633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "5fa746eed6584d6f8b4f85dac7771686"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "660a880d7f52010aa86498bd7208619b"
server_key_iv: "c6b9162f85ff3ffe7ed711fd9b014bdb"
server_key_version: "encryptionkey@0"
I20260805 23:05:02.415328 4633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:02.434588 4633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:02.437868 4633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:02.438043 4633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:02.446975 4633 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:41011
I20260805 23:05:02.447012 4741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:41011 every 8 connection(s)
I20260805 23:05:02.448055 4633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:02.451125 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:02.453967 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 4633
I20260805 23:05:02.454072 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:02.454347 3121 external_mini_cluster.cc:1428] Setting key 4c20a22755782b20824eb29758224bb1
I20260805 23:05:02.458264 4742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686: Bootstrap starting.
I20260805 23:05:02.461156 4742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:02.462009 4742 log.cc:824] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:05:02 dist-test-slave-2d19 krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:02.464726 4742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686: No bootstrap required, opened a new log
I20260805 23:05:02.469712 4742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa746eed6584d6f8b4f85dac7771686" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41011 } }
I20260805 23:05:02.470042 4742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:02.470111 4742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa746eed6584d6f8b4f85dac7771686, State: Initialized, Role: FOLLOWER
I20260805 23:05:02.470649 4742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa746eed6584d6f8b4f85dac7771686" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41011 } }
I20260805 23:05:02.470831 4742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:02.470896 4742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:02.470939 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:02.455829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.3.12.126:41011)
0805 23:05:02.456323 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:02.456343 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:02.456392 (+ 49us) server_negotiation.cc:407] Connection header received
0805 23:05:02.457246 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:02.457273 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:02.457711 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:02.458095 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:02.458898 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.460481 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.461443 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.461965 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.462167 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:02.465234 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:02.465244 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:02.465267 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:02.465312 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:02.467355 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:02.467855 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:02.467862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:02.467869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:02.468036 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:02.468401 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:02.468404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:02.468406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:02.468987 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:02.469224 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:02.469527 (+ 303us) server_negotiation.cc:299] Negotiation successful
0805 23:05:02.469863 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":121,"threads_started":1}
I20260805 23:05:02.470991 4742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:02.472157 4742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa746eed6584d6f8b4f85dac7771686" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41011 } }
I20260805 23:05:02.472620 4742 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa746eed6584d6f8b4f85dac7771686; no voters:
I20260805 23:05:02.473055 4742 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:02.473193 4747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:02.473984 4747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa746eed6584d6f8b4f85dac7771686, State: Running, Role: LEADER
I20260805 23:05:02.474363 4742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:02.474346 4747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa746eed6584d6f8b4f85dac7771686" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41011 } }
I20260805 23:05:02.476438 4748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa746eed6584d6f8b4f85dac7771686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa746eed6584d6f8b4f85dac7771686" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41011 } } }
I20260805 23:05:02.476629 4748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:02.476985 4749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa746eed6584d6f8b4f85dac7771686. Latest consensus state: current_term: 1 leader_uuid: "5fa746eed6584d6f8b4f85dac7771686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa746eed6584d6f8b4f85dac7771686" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41011 } } }
I20260805 23:05:02.477205 4749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686 [sys.catalog]: This master's current role is: LEADER
W20260805 23:05:02.479593 4770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:05:02.479696 4770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:05:02.479772 4772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:02.480612 4772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:02.487500 4772 catalog_manager.cc:1347] Generated new cluster ID: 84f1c43b8cd44d62aa5d56fafbf43f78
I20260805 23:05:02.487613 4772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:02.513911 4772 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:02.515357 4772 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:02.524066 4772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686: Generated new TSK 0
I20260805 23:05:02.524968 4772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:05:02.572757 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41011
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:02.712556 4779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:02.712848 4779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:02.712898 4779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:02.716964 4779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:02.717031 4779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:02.717109 4779 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:02.717180 4779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:02.722783 4779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:41011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:02.723970 4779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:02.725064 4779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:02.735463 4789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:02.735591 4779 server_base.cc:1034] running on GCE node
W20260805 23:05:02.735514 4785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:02.735541 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:02.736243 4779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:02.736946 4779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:02.738173 4779 hybrid_clock.cc:648] HybridClock initialized: now 1785971102738152 us; error 31 us; skew 500 ppm
Aug 05 23:05:02 dist-test-slave-2d19 krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971102, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:02.741271 4779 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:05:02.742259 4779 webserver.cc:460] Webserver started at http://127.3.12.65:43103/ using document root <none> and password file <none>
I20260805 23:05:02.742858 4779 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:02.742955 4779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:02.743204 4779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:02.745357 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "3a2afcfa1adc4efebf29e69fc24326ad"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "829689543f5b525cbcb4e7a419f32762"
server_key_iv: "3d6f48df4c8f02f58e5e3f1bc67aee7b"
server_key_version: "encryptionkey@0"
I20260805 23:05:02.746011 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "3a2afcfa1adc4efebf29e69fc24326ad"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "829689543f5b525cbcb4e7a419f32762"
server_key_iv: "3d6f48df4c8f02f58e5e3f1bc67aee7b"
server_key_version: "encryptionkey@0"
I20260805 23:05:02.749889 4779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:05:02.752548 4798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:02.753789 4779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:02.753934 4779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "3a2afcfa1adc4efebf29e69fc24326ad"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "829689543f5b525cbcb4e7a419f32762"
server_key_iv: "3d6f48df4c8f02f58e5e3f1bc67aee7b"
server_key_version: "encryptionkey@0"
I20260805 23:05:02.754053 4779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:02.792217 4779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:02.795837 4779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:02.796054 4779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:02.797663 4779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:02.797751 4779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:02.797824 4779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:02.797881 4779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:02 dist-test-slave-2d19 krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971102, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:02.810114 4779 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:46561
I20260805 23:05:02.810187 4964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:46561 every 8 connection(s)
I20260805 23:05:02.811301 4779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:02.812886 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:02.798724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:57485 (local address 127.3.12.126:41011)
0805 23:05:02.798855 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:02.798859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:02.800114 (+ 1255us) server_negotiation.cc:407] Connection header received
0805 23:05:02.801319 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:02.801326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:02.801385 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:02.801442 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:02.803267 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.803813 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.805242 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.805467 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.805595 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:02.808610 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:02.808615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:02.808620 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:02.808660 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:02.810454 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:02.811027 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:02.811030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:02.811033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:02.811131 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:02.811492 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:02.811496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:02.811498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:02.811764 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:02.811871 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:02.812643 (+ 772us) server_negotiation.cc:299] Negotiation successful
0805 23:05:02.812763 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260805 23:05:02.813016 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 4779
I20260805 23:05:02.813127 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:02.813405 3121 external_mini_cluster.cc:1428] Setting key a8bca37e15717876969ecd8e33d90d48
I20260805 23:05:02.813987 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:02.799340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41011 (local address 127.3.12.65:57485)
0805 23:05:02.799893 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:02.799940 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:02.801055 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:02.801859 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:02.801870 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:02.802417 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:02.803030 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:02.803045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.803957 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.803960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:02.805094 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:02.805104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.805649 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.805654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:02.805816 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:02.806461 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:02.806477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:02.808395 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:02.810575 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:02.810581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:02.810592 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:02.810886 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:02.811232 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:02.811234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:02.811235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:02.811372 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:02.811943 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:02.811949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:02.812356 (+ 407us) client_negotiation.cc:769] Sending connection context
0805 23:05:02.812563 (+ 207us) client_negotiation.cc:240] Negotiation successful
0805 23:05:02.812927 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":151,"threads_started":1}
I20260805 23:05:02.815456 4965 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41011
I20260805 23:05:02.815624 4965 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:02.816284 4965 heartbeater.cc:511] Master 127.3.12.126:41011 requested a full tablet report, sending...
I20260805 23:05:02.818177 4677 ts_manager.cc:194] Registered new tserver with Master: 3a2afcfa1adc4efebf29e69fc24326ad (127.3.12.65:46561)
I20260805 23:05:02.819859 4677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:57485
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:02.827775 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:02.817676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41011 (local address 127.3.12.65:59139)
0805 23:05:02.820446 (+ 2770us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:02.820460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:02.820611 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:02.820871 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:02.820874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:02.821110 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:02.821206 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:02.821211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.821968 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.821971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:02.822520 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:02.822529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.822982 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.822985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:02.823087 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:02.823640 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:02.823652 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:02.823977 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:02.826086 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:02.826090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:02.826092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:02.826315 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:02.826656 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:02.826659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:02.826662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:02.826758 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:02.827177 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:02.827180 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:02.827284 (+ 104us) client_negotiation.cc:769] Sending connection context
0805 23:05:02.827397 (+ 113us) client_negotiation.cc:240] Negotiation successful
0805 23:05:02.827571 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2689,"thread_start_us":126,"threads_started":1}
I20260805 23:05:02.827803 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:02.817742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:59139 (local address 127.3.12.126:41011)
0805 23:05:02.817884 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:02.817889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:02.820561 (+ 2672us) server_negotiation.cc:407] Connection header received
0805 23:05:02.820718 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:02.820727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:02.820767 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:02.820854 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:02.821323 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.821830 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.822642 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:02.822865 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:02.822974 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:02.824123 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:02.824126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:02.824129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:02.824161 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:02.825889 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:02.826428 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:02.826433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:02.826435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:02.826555 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:02.826857 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:02.826862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:02.826863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:02.827064 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:02.827178 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:02.827455 (+ 277us) server_negotiation.cc:299] Negotiation successful
0805 23:05:02.827571 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:02.847910 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41011
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:02.968963 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:02.969228 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:02.969306 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:02.974149 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:02.974221 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:02.974295 4977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:02.974391 4977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:02.980693 4977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:41011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:02.981904 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:02.983086 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:02.993757 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:02.993943 4977 server_base.cc:1034] running on GCE node
W20260805 23:05:02.993752 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:02.993749 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:02.994547 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:02.995278 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:02.996456 4977 hybrid_clock.cc:648] HybridClock initialized: now 1785971102996433 us; error 34 us; skew 500 ppm
Aug 05 23:05:02 dist-test-slave-2d19 krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971102, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:02.999447 4977 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:03.000432 4977 webserver.cc:460] Webserver started at http://127.3.12.66:40161/ using document root <none> and password file <none>
I20260805 23:05:03.000986 4977 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:03.001045 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:03.001231 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:03.003278 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "7d038f40da6190af3497d9b95702fca0"
server_key_iv: "b5d3ab5c952dcf13e945f98b4e857ff3"
server_key_version: "encryptionkey@0"
I20260805 23:05:03.003813 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "7d038f40da6190af3497d9b95702fca0"
server_key_iv: "b5d3ab5c952dcf13e945f98b4e857ff3"
server_key_version: "encryptionkey@0"
I20260805 23:05:03.007742 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260805 23:05:03.010310 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:03.011580 4977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:03.011710 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf"
format_stamp: "Formatted at 2026-08-05 23:05:02 on dist-test-slave-2d19"
server_key: "7d038f40da6190af3497d9b95702fca0"
server_key_iv: "b5d3ab5c952dcf13e945f98b4e857ff3"
server_key_version: "encryptionkey@0"
I20260805 23:05:03.011818 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:03.054629 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:03.058447 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:03.058683 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:03.060295 4977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:03.060372 4977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:03.060439 4977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:03.060493 4977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:03 dist-test-slave-2d19 krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971102, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:03.071928 4977 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:38663
I20260805 23:05:03.071995 5153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:38663 every 8 connection(s)
I20260805 23:05:03.072970 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:03.076165 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.061382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:38783 (local address 127.3.12.126:41011)
0805 23:05:03.061559 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.061564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.062500 (+ 936us) server_negotiation.cc:407] Connection header received
0805 23:05:03.063788 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.063791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.063855 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.063969 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:03.065872 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.066768 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.068538 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.068889 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.069010 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.072144 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:03.072148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:03.072151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:03.072189 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:03.074045 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.074541 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.074544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.074545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.074647 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.075039 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.075042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.075043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.075241 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:03.075326 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.075914 (+ 588us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.076035 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260805 23:05:03.076850 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 4977
I20260805 23:05:03.076975 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:03.077260 3121 external_mini_cluster.cc:1428] Setting key 5729a56af04bba851ebdf3937d28d68a
I20260805 23:05:03.077245 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.061833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41011 (local address 127.3.12.66:38783)
0805 23:05:03.062326 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.062381 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.063551 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.064229 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.064244 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.064818 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:03.065629 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.065668 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.066986 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.066992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.068370 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.068381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.069068 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.069071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.069222 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.069958 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:03.069984 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:03.071881 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:03.074173 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.074178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.074190 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.074448 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.074806 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.074808 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.074810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.074948 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.075338 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:03.075343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:03.075592 (+ 249us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.075854 (+ 262us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.076140 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":145,"threads_started":1}
I20260805 23:05:03.078486 5155 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41011
I20260805 23:05:03.078665 5155 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:03.079253 5155 heartbeater.cc:511] Master 127.3.12.126:41011 requested a full tablet report, sending...
I20260805 23:05:03.080900 4675 ts_manager.cc:194] Registered new tserver with Master: 0140a7c7cf7d46e6aa8b352888c9b5bf (127.3.12.66:38663)
I20260805 23:05:03.081771 4675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:38783
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:05:03.089980 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.080833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41011 (local address 127.3.12.66:57403)
0805 23:05:03.081124 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.081142 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.081219 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.082437 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.082440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.082671 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:03.082855 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.082863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.083968 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.083972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.084644 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.084654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.085316 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.085318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.085387 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.085964 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:03.085975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:03.086314 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:03.088359 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.088362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.088364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.088564 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.088878 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.088880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.088881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.088958 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.089514 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:03.089517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:03.089605 (+ 88us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.089698 (+ 93us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.089826 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":20,"thread_start_us":108,"threads_started":1}
I20260805 23:05:03.089986 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.082120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:57403 (local address 127.3.12.126:41011)
0805 23:05:03.082243 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.082246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.082257 (+ 11us) server_negotiation.cc:407] Connection header received
0805 23:05:03.082288 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.082290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.082324 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.082415 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:03.082994 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.083818 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.084766 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.085186 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.085313 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.086462 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:03.086466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:03.086468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:03.086497 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:03.088183 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.088672 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.088678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.088682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.088774 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.089049 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.089052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.089055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.089396 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:03.089518 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.089772 (+ 254us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.089871 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:03.112411 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41011
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:03.233835 5166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:03.234284 5166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:03.234392 5166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:03.239151 5166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:03.239266 5166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:03.239339 5166 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:03.239413 5166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:03.245393 5166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:41011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:03.246562 5166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:03.247619 5166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:03.257921 5171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:03.257939 5175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:03.257969 5172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:03.258278 5166 server_base.cc:1034] running on GCE node
I20260805 23:05:03.258693 5166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:03.259356 5166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:03.260519 5166 hybrid_clock.cc:648] HybridClock initialized: now 1785971103260470 us; error 61 us; skew 500 ppm
Aug 05 23:05:03 dist-test-slave-2d19 krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971103, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:03.263535 5166 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:03.264561 5166 webserver.cc:460] Webserver started at http://127.3.12.67:34537/ using document root <none> and password file <none>
I20260805 23:05:03.265203 5166 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:03.265302 5166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:03.265547 5166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:03.267644 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "f663865ada3045238fc5adc3d0936862"
format_stamp: "Formatted at 2026-08-05 23:05:03 on dist-test-slave-2d19"
server_key: "e0bb6b70cff774193c926285545c634a"
server_key_iv: "0ccb395a449259b0b33bc3283cc001f4"
server_key_version: "encryptionkey@0"
I20260805 23:05:03.268227 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "f663865ada3045238fc5adc3d0936862"
format_stamp: "Formatted at 2026-08-05 23:05:03 on dist-test-slave-2d19"
server_key: "e0bb6b70cff774193c926285545c634a"
server_key_iv: "0ccb395a449259b0b33bc3283cc001f4"
server_key_version: "encryptionkey@0"
I20260805 23:05:03.272243 5166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 23:05:03.274904 5183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:03.276492 5166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:03.276620 5166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "f663865ada3045238fc5adc3d0936862"
format_stamp: "Formatted at 2026-08-05 23:05:03 on dist-test-slave-2d19"
server_key: "e0bb6b70cff774193c926285545c634a"
server_key_iv: "0ccb395a449259b0b33bc3283cc001f4"
server_key_version: "encryptionkey@0"
I20260805 23:05:03.276747 5166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:03.290544 5166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:03.293641 5166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:03.293860 5166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:03.295363 5166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:03.295423 5166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:03.295504 5166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:03.295542 5166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:03 dist-test-slave-2d19 krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971103, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:03.307229 5166 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:42171
I20260805 23:05:03.307283 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:42171 every 8 connection(s)
I20260805 23:05:03.308501 5166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:03.310678 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 5166
I20260805 23:05:03.310811 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:03.311102 3121 external_mini_cluster.cc:1428] Setting key ca91415ae5dd5e3316b848af7e764960
I20260805 23:05:03.311149 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.296387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:39027 (local address 127.3.12.126:41011)
0805 23:05:03.296525 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.296529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.297730 (+ 1201us) server_negotiation.cc:407] Connection header received
0805 23:05:03.298859 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.298883 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.298954 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.299044 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:03.300869 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.301596 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.303620 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.304012 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.304123 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.307211 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:03.307215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:03.307218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:03.307250 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:03.308898 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.309519 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.309521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.309523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.309645 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.310028 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.310032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.310034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.310225 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:03.310337 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.310839 (+ 502us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.310977 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260805 23:05:03.312166 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.297001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41011 (local address 127.3.12.67:39027)
0805 23:05:03.297560 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.297605 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.298566 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.299576 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.299586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.300039 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:03.300705 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.300721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.301764 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.301768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.303469 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.303483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.304176 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.304181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.304411 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.305051 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:03.305079 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:03.307002 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:03.309032 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.309038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.309053 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.309417 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.309764 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.309767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.309770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.309916 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.310337 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:03.310344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:03.310592 (+ 248us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.310779 (+ 187us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.311107 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":156,"threads_started":1}
I20260805 23:05:03.313501 5359 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41011
I20260805 23:05:03.313684 5359 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:03.314211 5359 heartbeater.cc:511] Master 127.3.12.126:41011 requested a full tablet report, sending...
I20260805 23:05:03.315312 4675 ts_manager.cc:194] Registered new tserver with Master: f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171)
I20260805 23:05:03.316080 4675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:39027
I20260805 23:05:03.322753 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.315736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41011 (local address 127.3.12.67:44549)
0805 23:05:03.316149 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.316163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.316293 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.316596 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.316598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.316892 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:03.317030 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.317039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.317785 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.317788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.318578 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.318596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.319050 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.319052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.319120 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.319555 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:03.319566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:03.319913 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:03.321520 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.321523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.321525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.321720 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.321963 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.321965 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.321966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.322043 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.322352 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:03.322354 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:03.322426 (+ 72us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.322514 (+ 88us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.322614 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260805 23:05:03.322780 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.315852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:44549 (local address 127.3.12.126:41011)
0805 23:05:03.315980 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.315985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.316295 (+ 310us) server_negotiation.cc:407] Connection header received
0805 23:05:03.316441 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.316444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.316494 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.316582 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:03.317147 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.317629 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.318696 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.318950 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.319030 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.320054 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:03.320058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:03.320060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:03.320081 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:03.321411 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.321806 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.321808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.321810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.321880 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.322119 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.322121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.322122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.322272 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:03.322347 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.322578 (+ 231us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.322673 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260805 23:05:03.325523 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 23:05:03 dist-test-slave-2d19 krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 23:05:03 dist-test-slave-2d19 krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:03.352905 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.344442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.3.12.126:41011)
0805 23:05:03.344592 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.344595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.344608 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:05:03.344658 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.344661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.344704 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.344806 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:03.345540 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.346076 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.346946 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.347165 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.347301 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.349816 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:03.349819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:03.349822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:03.349849 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:03.351127 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.351574 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.351578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.351579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.351664 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.351976 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.351979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.351981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.352175 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:03.352355 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.352598 (+ 243us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.352733 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:05:03.355870 4675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:03.359057 4675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:03.373831 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.367626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:46561 (local address 127.0.0.1:47862)
0805 23:05:03.368375 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.368433 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.368572 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.369295 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.369301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.369333 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:03.369559 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.369567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.371093 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.371097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.372365 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.372375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.373231 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.373235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.373339 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.373376 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.373530 (+ 154us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.373653 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":121,"spinlock_wait_cycles":27648,"thread_start_us":112,"threads_started":1}
I20260805 23:05:03.373965 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.368341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38663 (local address 127.0.0.1:38132)
0805 23:05:03.368843 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.368859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.368956 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.369724 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.369727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.369747 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:03.369839 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.369846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.370804 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.370809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.372525 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.372533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.373542 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.373545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.373624 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.373654 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.373729 (+ 75us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.373785 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":59,"thread_start_us":90,"threads_started":1}
I20260805 23:05:03.374027 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.368777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.3.12.66:38663)
0805 23:05:03.369203 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.369216 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.369238 (+ 22us) server_negotiation.cc:407] Connection header received
0805 23:05:03.369312 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.369317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.369516 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.369757 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:03.369962 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.370683 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.372644 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.373413 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.373526 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.373666 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.373759 (+ 93us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.373862 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":119,"threads_started":1}
I20260805 23:05:03.374333 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.368055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.3.12.65:46561)
0805 23:05:03.368854 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.368861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.368882 (+ 21us) server_negotiation.cc:407] Connection header received
0805 23:05:03.368948 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.368964 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.369130 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.369711 (+ 581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:03.369852 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.370970 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.372506 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.373099 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.373231 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.373389 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.374085 (+ 696us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.374204 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":86,"threads_started":1}
I20260805 23:05:03.374615 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.367566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:42171 (local address 127.0.0.1:34134)
0805 23:05:03.368375 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.368433 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.368572 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.369308 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.369310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.369333 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:03.369562 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.369567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.371139 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.371142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.372349 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.372361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.373687 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.373693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.373785 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.373809 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.373881 (+ 72us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.374413 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":173,"spinlock_wait_cycles":7808,"thread_start_us":85,"threads_started":1}
I20260805 23:05:03.375161 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.367695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.3.12.67:42171)
0805 23:05:03.368506 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.368519 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.368546 (+ 27us) server_negotiation.cc:407] Connection header received
0805 23:05:03.368783 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.368789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.368999 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.369150 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:03.369696 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.370970 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.372488 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.373322 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.373435 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.373613 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.374904 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.374994 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":114,"threads_started":1}
I20260805 23:05:03.377267 4871 tablet_service.cc:1467] Processing CreateTablet for tablet bfb38b030c754c69ab823218376b6aef (DEFAULT_TABLE table=test-table [id=2b02b81a75cc43efa14864287cb0e771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:03.377382 5066 tablet_service.cc:1467] Processing CreateTablet for tablet bfb38b030c754c69ab823218376b6aef (DEFAULT_TABLE table=test-table [id=2b02b81a75cc43efa14864287cb0e771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:03.377663 5262 tablet_service.cc:1467] Processing CreateTablet for tablet bfb38b030c754c69ab823218376b6aef (DEFAULT_TABLE table=test-table [id=2b02b81a75cc43efa14864287cb0e771]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:03.378448 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb38b030c754c69ab823218376b6aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.378481 5066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb38b030c754c69ab823218376b6aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.378597 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb38b030c754c69ab823218376b6aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.385281 5409 tablet_bootstrap.cc:492] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Bootstrap starting.
I20260805 23:05:03.387362 5412 tablet_bootstrap.cc:492] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Bootstrap starting.
I20260805 23:05:03.387764 5409 tablet_bootstrap.cc:654] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.388701 5409 log.cc:824] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:03.389357 5410 tablet_bootstrap.cc:492] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Bootstrap starting.
I20260805 23:05:03.389420 5412 tablet_bootstrap.cc:654] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.390273 5412 log.cc:824] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:03.391217 5409 tablet_bootstrap.cc:492] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: No bootstrap required, opened a new log
I20260805 23:05:03.391443 5409 ts_tablet_manager.cc:1397] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:05:03.391595 5410 tablet_bootstrap.cc:654] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.392377 5410 log.cc:824] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:03.394214 5412 tablet_bootstrap.cc:492] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: No bootstrap required, opened a new log
I20260805 23:05:03.394232 5410 tablet_bootstrap.cc:492] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: No bootstrap required, opened a new log
I20260805 23:05:03.394425 5412 ts_tablet_manager.cc:1397] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260805 23:05:03.394438 5410 ts_tablet_manager.cc:1397] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 23:05:03.396656 5409 raft_consensus.cc:348] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.396965 5409 raft_consensus.cc:374] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.397022 5409 raft_consensus.cc:729] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2afcfa1adc4efebf29e69fc24326ad, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.397598 5409 consensus_queue.cc:260] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.398427 5410 raft_consensus.cc:348] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.398692 5409 ts_tablet_manager.cc:1428] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260805 23:05:03.398692 5410 raft_consensus.cc:374] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.398783 5410 raft_consensus.cc:729] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0140a7c7cf7d46e6aa8b352888c9b5bf, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.398772 5412 raft_consensus.cc:348] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.399034 5412 raft_consensus.cc:374] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.399092 5412 raft_consensus.cc:729] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f663865ada3045238fc5adc3d0936862, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.399130 4965 heartbeater.cc:503] Master 127.3.12.126:41011 was elected leader, sending a full tablet report...
I20260805 23:05:03.399324 5410 consensus_queue.cc:260] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.399518 5412 consensus_queue.cc:260] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.400058 5155 heartbeater.cc:503] Master 127.3.12.126:41011 was elected leader, sending a full tablet report...
I20260805 23:05:03.400408 5410 ts_tablet_manager.cc:1428] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 23:05:03.400523 5412 ts_tablet_manager.cc:1428] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260805 23:05:03.400892 5359 heartbeater.cc:503] Master 127.3.12.126:41011 was elected leader, sending a full tablet report...
W20260805 23:05:03.559744 5363 tablet.cc:2369] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 23:05:03.563042 4967 tablet.cc:2369] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 23:05:03.574477 5156 tablet.cc:2369] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:05:03.672210 5418 raft_consensus.cc:482] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:03.672626 5418 raft_consensus.cc:504] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.673990 5418 leader_election.cc:288] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171), 3a2afcfa1adc4efebf29e69fc24326ad (127.3.12.65:46561)
I20260805 23:05:03.679875 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.674424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:49997 (local address 127.3.12.67:42171)
0805 23:05:03.674583 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.674588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.674737 (+ 149us) server_negotiation.cc:407] Connection header received
0805 23:05:03.674934 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.674936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.675008 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.675099 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:03.675601 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.676533 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.678446 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.679144 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.679268 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.679340 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.679645 (+ 305us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.679733 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260805 23:05:03.679908 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.674319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:46561 (local address 127.3.12.66:55719)
0805 23:05:03.674800 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.674817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.674971 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.675376 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.675379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.675392 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:03.675481 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.675489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.676681 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.676683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.678265 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.678279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.679298 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.679302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.679492 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.679535 (+ 43us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.679610 (+ 75us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.679747 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":181,"thread_start_us":126,"threads_started":1}
I20260805 23:05:03.679875 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.674571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:55719 (local address 127.3.12.65:46561)
0805 23:05:03.674693 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.674700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.674890 (+ 190us) server_negotiation.cc:407] Connection header received
0805 23:05:03.675173 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.675175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.675224 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.675333 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:03.675601 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.676585 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.678446 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.679147 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.679268 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.679340 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.679645 (+ 305us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.679733 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260805 23:05:03.679910 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.674233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:42171 (local address 127.3.12.66:49997)
0805 23:05:03.674565 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.674584 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.674770 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.675214 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.675237 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.675261 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:03.675400 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.675407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.676679 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.676682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.678265 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.678279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.679298 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.679302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.679496 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.679523 (+ 27us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.679610 (+ 87us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.679723 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":197,"thread_start_us":100,"threads_started":1}
I20260805 23:05:03.680789 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb38b030c754c69ab823218376b6aef" candidate_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" is_pre_election: true
I20260805 23:05:03.681109 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb38b030c754c69ab823218376b6aef" candidate_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f663865ada3045238fc5adc3d0936862" is_pre_election: true
I20260805 23:05:03.681221 4899 raft_consensus.cc:2454] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0140a7c7cf7d46e6aa8b352888c9b5bf in term 0.
I20260805 23:05:03.681478 5293 raft_consensus.cc:2454] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0140a7c7cf7d46e6aa8b352888c9b5bf in term 0.
I20260805 23:05:03.681845 4999 leader_election.cc:302] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0140a7c7cf7d46e6aa8b352888c9b5bf, 3a2afcfa1adc4efebf29e69fc24326ad; no voters:
I20260805 23:05:03.682199 5418 raft_consensus.cc:2781] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:03.682287 5418 raft_consensus.cc:482] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:03.682323 5418 raft_consensus.cc:3037] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:03.683570 5418 raft_consensus.cc:504] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.684075 5418 leader_election.cc:288] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [CANDIDATE]: Term 1 election: Requested vote from peers f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171), 3a2afcfa1adc4efebf29e69fc24326ad (127.3.12.65:46561)
I20260805 23:05:03.684482 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb38b030c754c69ab823218376b6aef" candidate_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f663865ada3045238fc5adc3d0936862"
I20260805 23:05:03.684638 5293 raft_consensus.cc:3037] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:03.684736 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb38b030c754c69ab823218376b6aef" candidate_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2afcfa1adc4efebf29e69fc24326ad"
I20260805 23:05:03.684929 4899 raft_consensus.cc:3037] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:03.686196 4899 raft_consensus.cc:2454] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0140a7c7cf7d46e6aa8b352888c9b5bf in term 1.
I20260805 23:05:03.686604 4999 leader_election.cc:302] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0140a7c7cf7d46e6aa8b352888c9b5bf, 3a2afcfa1adc4efebf29e69fc24326ad; no voters:
I20260805 23:05:03.686857 5418 raft_consensus.cc:2781] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:03.686866 5293 raft_consensus.cc:2454] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0140a7c7cf7d46e6aa8b352888c9b5bf in term 1.
I20260805 23:05:03.687100 5418 raft_consensus.cc:686] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 LEADER]: Becoming Leader. State: Replica: 0140a7c7cf7d46e6aa8b352888c9b5bf, State: Running, Role: LEADER
I20260805 23:05:03.687479 5418 consensus_queue.cc:237] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } }
I20260805 23:05:03.690603 4677 catalog_manager.cc:5495] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0140a7c7cf7d46e6aa8b352888c9b5bf (127.3.12.66). New cstate: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } health_report { overall_health: HEALTHY } } }
I20260805 23:05:03.745496 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.741465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.3.12.66:38663)
0805 23:05:03.741631 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.741636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.741661 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:05:03.742035 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.742041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.742110 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.742200 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:03.742497 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.743044 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.743958 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.744182 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.744326 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.744687 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:03.745127 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:03.745242 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.745322 (+ 80us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.745381 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260805 23:05:03.751927 4899 raft_consensus.cc:1261] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Refusing update from remote peer 0140a7c7cf7d46e6aa8b352888c9b5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:03.751926 5293 raft_consensus.cc:1261] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Refusing update from remote peer 0140a7c7cf7d46e6aa8b352888c9b5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:03.752768 5418 consensus_queue.cc:1035] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:03.753021 5422 consensus_queue.cc:1035] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:03.763428 5427 mvcc.cc:205] Tried to move back new op lower bound from 7315337640960045056 to 7315337640706387968. Current Snapshot: MvccSnapshot[applied={T|T < 7315337640960045056}]
I20260805 23:05:03.773412 4677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:03.773643 4677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:03.776551 4677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686: Sending DeleteTablet for 3 replicas of tablet bfb38b030c754c69ab823218376b6aef
I20260805 23:05:03.777486 5262 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb38b030c754c69ab823218376b6aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:03 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:34134
I20260805 23:05:03.777661 5066 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb38b030c754c69ab823218376b6aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:03 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:38132
I20260805 23:05:03.777671 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet bfb38b030c754c69ab823218376b6aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:03 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:47862
I20260805 23:05:03.778093 5438 tablet_replica.cc:330] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: stopping tablet replica
I20260805 23:05:03.778422 5439 tablet_replica.cc:330] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: stopping tablet replica
I20260805 23:05:03.778576 5438 raft_consensus.cc:2229] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:03.778792 5439 raft_consensus.cc:2229] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:03.779122 5438 raft_consensus.cc:2258] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:03.779084 4677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260805 23:05:03.779215 5439 raft_consensus.cc:2258] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Raft consensus is shut down!
W20260805 23:05:03.779752 4677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:03.779806 5441 tablet_replica.cc:330] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: stopping tablet replica
I20260805 23:05:03.780148 5441 raft_consensus.cc:2229] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:03.780987 5441 raft_consensus.cc:2258] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:03.781523 5439 ts_tablet_manager.cc:1905] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:03.783563 5441 ts_tablet_manager.cc:1905] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:03.784799 5439 ts_tablet_manager.cc:1918] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:03.784929 5439 log.cc:1196] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/bfb38b030c754c69ab823218376b6aef
I20260805 23:05:03.786626 5438 ts_tablet_manager.cc:1905] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:03.787012 5262 tablet_service.cc:1467] Processing CreateTablet for tablet b57104affe9947a985ac0447e1aed0ed (DEFAULT_TABLE table=test-table [id=fd3ed7241a994966a6cbd6ae1b3c07d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:03.787277 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57104affe9947a985ac0447e1aed0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.787438 5441 ts_tablet_manager.cc:1918] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:03.787537 5441 log.cc:1196] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/bfb38b030c754c69ab823218376b6aef
I20260805 23:05:03.787912 5441 ts_tablet_manager.cc:1939] T bfb38b030c754c69ab823218376b6aef P 0140a7c7cf7d46e6aa8b352888c9b5bf: Deleting consensus metadata
I20260805 23:05:03.789764 5438 ts_tablet_manager.cc:1918] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:03.789861 5438 log.cc:1196] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/bfb38b030c754c69ab823218376b6aef
I20260805 23:05:03.789960 5412 tablet_bootstrap.cc:492] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: Bootstrap starting.
I20260805 23:05:03.790218 5438 ts_tablet_manager.cc:1939] T bfb38b030c754c69ab823218376b6aef P f663865ada3045238fc5adc3d0936862: Deleting consensus metadata
I20260805 23:05:03.790761 4655 catalog_manager.cc:4841] TS 0140a7c7cf7d46e6aa8b352888c9b5bf (127.3.12.66:38663): tablet bfb38b030c754c69ab823218376b6aef (table test-table [id=2b02b81a75cc43efa14864287cb0e771]) successfully deleted
I20260805 23:05:03.791044 5439 ts_tablet_manager.cc:1939] T bfb38b030c754c69ab823218376b6aef P 3a2afcfa1adc4efebf29e69fc24326ad: Deleting consensus metadata
I20260805 23:05:03.791383 5412 tablet_bootstrap.cc:654] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.791356 4871 tablet_service.cc:1467] Processing CreateTablet for tablet b57104affe9947a985ac0447e1aed0ed (DEFAULT_TABLE table=test-table [id=fd3ed7241a994966a6cbd6ae1b3c07d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:03.791618 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57104affe9947a985ac0447e1aed0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.792336 4655 catalog_manager.cc:4841] TS 3a2afcfa1adc4efebf29e69fc24326ad (127.3.12.65:46561): tablet bfb38b030c754c69ab823218376b6aef (table test-table [id=2b02b81a75cc43efa14864287cb0e771]) successfully deleted
I20260805 23:05:03.793256 5412 tablet_bootstrap.cc:492] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: No bootstrap required, opened a new log
I20260805 23:05:03.793330 5412 ts_tablet_manager.cc:1397] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:03.794016 5412 raft_consensus.cc:348] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.794128 5412 raft_consensus.cc:374] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.794159 5412 raft_consensus.cc:729] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f663865ada3045238fc5adc3d0936862, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.794188 5409 tablet_bootstrap.cc:492] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: Bootstrap starting.
I20260805 23:05:03.794317 5412 consensus_queue.cc:260] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.794574 5412 ts_tablet_manager.cc:1428] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260805 23:05:03.795039 4657 catalog_manager.cc:4841] TS f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171): tablet bfb38b030c754c69ab823218376b6aef (table test-table [id=2b02b81a75cc43efa14864287cb0e771]) successfully deleted
I20260805 23:05:03.796115 5066 tablet_service.cc:1467] Processing CreateTablet for tablet b57104affe9947a985ac0447e1aed0ed (DEFAULT_TABLE table=test-table [id=fd3ed7241a994966a6cbd6ae1b3c07d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:03.796504 5066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57104affe9947a985ac0447e1aed0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.796706 5409 tablet_bootstrap.cc:654] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.798476 5409 tablet_bootstrap.cc:492] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: No bootstrap required, opened a new log
I20260805 23:05:03.798560 5409 ts_tablet_manager.cc:1397] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260805 23:05:03.799643 5409 raft_consensus.cc:348] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.799782 5409 raft_consensus.cc:374] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.799820 5410 tablet_bootstrap.cc:492] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: Bootstrap starting.
I20260805 23:05:03.799824 5409 raft_consensus.cc:729] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2afcfa1adc4efebf29e69fc24326ad, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.800060 5409 consensus_queue.cc:260] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.800549 5409 ts_tablet_manager.cc:1428] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:03.801792 5410 tablet_bootstrap.cc:654] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.804090 5410 tablet_bootstrap.cc:492] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: No bootstrap required, opened a new log
I20260805 23:05:03.804191 5410 ts_tablet_manager.cc:1397] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260805 23:05:03.804944 5410 raft_consensus.cc:348] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.805073 5410 raft_consensus.cc:374] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.805105 5410 raft_consensus.cc:729] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0140a7c7cf7d46e6aa8b352888c9b5bf, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.805222 5410 consensus_queue.cc:260] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.805562 5410 ts_tablet_manager.cc:1428] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260805 23:05:03.884869 5417 raft_consensus.cc:482] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:03.885092 5417 raft_consensus.cc:504] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.886282 5417 leader_election.cc:288] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0140a7c7cf7d46e6aa8b352888c9b5bf (127.3.12.66:38663), f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171)
I20260805 23:05:03.891714 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.886869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:36591 (local address 127.3.12.67:42171)
0805 23:05:03.887025 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.887034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.887395 (+ 361us) server_negotiation.cc:407] Connection header received
0805 23:05:03.887633 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.887636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.887713 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.887800 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:03.888167 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.888907 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.890370 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.890912 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.891043 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.891117 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.891452 (+ 335us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.891548 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 23:05:03.891746 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.886547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:42171 (local address 127.3.12.65:36591)
0805 23:05:03.887289 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.887305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.887488 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.887824 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.887826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.887845 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:03.888013 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.888023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.889099 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.889102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.890223 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.890235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.891070 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.891074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.891266 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.891295 (+ 29us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.891428 (+ 133us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.891509 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":68,"thread_start_us":119,"threads_started":1}
I20260805 23:05:03.892457 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.886939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:51527 (local address 127.3.12.66:38663)
0805 23:05:03.887102 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.887107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.887147 (+ 40us) server_negotiation.cc:407] Connection header received
0805 23:05:03.887230 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.887233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.887291 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.887412 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:03.887760 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.888571 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.890781 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.891408 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.891534 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.891597 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.892183 (+ 586us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.892302 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 23:05:03.892555 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57104affe9947a985ac0447e1aed0ed" candidate_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f663865ada3045238fc5adc3d0936862" is_pre_election: true
I20260805 23:05:03.892788 5293 raft_consensus.cc:2454] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2afcfa1adc4efebf29e69fc24326ad in term 0.
I20260805 23:05:03.893011 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.886538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38663 (local address 127.3.12.65:51527)
0805 23:05:03.886923 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.886940 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.887068 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.887430 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.887433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.887455 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:03.887614 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.887621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.888721 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.888725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.890551 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.890579 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.891666 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.891670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.891979 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.892026 (+ 47us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.892644 (+ 618us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.892732 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":13,"spinlock_wait_cycles":3968,"thread_start_us":100,"threads_started":1}
I20260805 23:05:03.893319 4805 leader_election.cc:302] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2afcfa1adc4efebf29e69fc24326ad, f663865ada3045238fc5adc3d0936862; no voters:
I20260805 23:05:03.893680 5417 raft_consensus.cc:2781] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:03.893785 5417 raft_consensus.cc:482] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:03.893841 5417 raft_consensus.cc:3037] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:03.894466 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57104affe9947a985ac0447e1aed0ed" candidate_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" is_pre_election: true
I20260805 23:05:03.894804 5092 raft_consensus.cc:2454] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2afcfa1adc4efebf29e69fc24326ad in term 0.
I20260805 23:05:03.895659 5417 raft_consensus.cc:504] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.896106 5417 leader_election.cc:288] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [CANDIDATE]: Term 1 election: Requested vote from peers 0140a7c7cf7d46e6aa8b352888c9b5bf (127.3.12.66:38663), f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171)
I20260805 23:05:03.896656 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57104affe9947a985ac0447e1aed0ed" candidate_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf"
I20260805 23:05:03.896622 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57104affe9947a985ac0447e1aed0ed" candidate_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f663865ada3045238fc5adc3d0936862"
I20260805 23:05:03.896806 5293 raft_consensus.cc:3037] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:03.896814 5092 raft_consensus.cc:3037] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:03.898387 5293 raft_consensus.cc:2454] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2afcfa1adc4efebf29e69fc24326ad in term 1.
I20260805 23:05:03.898521 5092 raft_consensus.cc:2454] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2afcfa1adc4efebf29e69fc24326ad in term 1.
I20260805 23:05:03.899075 4801 leader_election.cc:302] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0140a7c7cf7d46e6aa8b352888c9b5bf, 3a2afcfa1adc4efebf29e69fc24326ad; no voters:
I20260805 23:05:03.899384 5417 raft_consensus.cc:2781] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:03.899706 5417 raft_consensus.cc:686] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 LEADER]: Becoming Leader. State: Replica: 3a2afcfa1adc4efebf29e69fc24326ad, State: Running, Role: LEADER
I20260805 23:05:03.900105 5417 consensus_queue.cc:237] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.903192 4677 catalog_manager.cc:5495] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a2afcfa1adc4efebf29e69fc24326ad (127.3.12.65). New cstate: current_term: 1 leader_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } health_report { overall_health: UNKNOWN } } }
Aug 05 23:05:03 dist-test-slave-2d19 krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971102, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:03.966424 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.954287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41011 (local address 127.0.0.1:35608)
0805 23:05:03.954746 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:03.954761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:03.954863 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:03.955358 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:03.955361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:03.955784 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:03.955923 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.955929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.957172 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.957175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.957897 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:03.957911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.959031 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.959034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:03.959108 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.959881 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:03.959895 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:03.961799 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:03.964490 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.964499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.964504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.964747 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.965228 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:03.965230 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:03.965232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:03.965316 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:03.965903 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:03.965909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:03.966049 (+ 140us) client_negotiation.cc:769] Sending connection context
0805 23:05:03.966147 (+ 98us) client_negotiation.cc:240] Negotiation successful
0805 23:05:03.966268 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1,"thread_start_us":172,"threads_started":1}
I20260805 23:05:03.966648 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:03.954423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.3.12.126:41011)
0805 23:05:03.954866 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:03.954909 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:03.954944 (+ 35us) server_negotiation.cc:407] Connection header received
0805 23:05:03.955018 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:03.955021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:03.955118 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:03.955330 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:03.956069 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.957019 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.958070 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:03.958749 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:03.959079 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:03.961959 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:03.961962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:03.961965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:03.961992 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:03.964347 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.964887 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.964891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.964893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.965072 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:03.965429 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:03.965435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:03.965437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:03.965778 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:03.965926 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:03.966254 (+ 328us) server_negotiation.cc:299] Negotiation successful
0805 23:05:03.966421 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":207,"threads_started":1}
I20260805 23:05:03.972505 4677 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:35608:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260805 23:05:03.973652 4677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:03.978996 4871 tablet_service.cc:1467] Processing CreateTablet for tablet 2333c324881e409f9a798e7dba89f04a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=247c25a400a94bb5a526f55089a7c51b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:03.979521 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2333c324881e409f9a798e7dba89f04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.979357 5262 tablet_service.cc:1467] Processing CreateTablet for tablet 2333c324881e409f9a798e7dba89f04a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=247c25a400a94bb5a526f55089a7c51b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:03.979822 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2333c324881e409f9a798e7dba89f04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.982542 5409 tablet_bootstrap.cc:492] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: Bootstrap starting.
I20260805 23:05:03.983032 5066 tablet_service.cc:1467] Processing CreateTablet for tablet 2333c324881e409f9a798e7dba89f04a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=247c25a400a94bb5a526f55089a7c51b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:03.983667 5066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2333c324881e409f9a798e7dba89f04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:03.983752 5412 tablet_bootstrap.cc:492] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: Bootstrap starting.
I20260805 23:05:03.984329 5409 tablet_bootstrap.cc:654] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.985164 5412 tablet_bootstrap.cc:654] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.986804 5412 tablet_bootstrap.cc:492] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: No bootstrap required, opened a new log
I20260805 23:05:03.986899 5412 ts_tablet_manager.cc:1397] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260805 23:05:03.987397 5410 tablet_bootstrap.cc:492] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: Bootstrap starting.
I20260805 23:05:03.987643 5412 raft_consensus.cc:348] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.987766 5412 raft_consensus.cc:374] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.987836 5412 raft_consensus.cc:729] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f663865ada3045238fc5adc3d0936862, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.988044 5412 consensus_queue.cc:260] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.988281 5410 tablet_bootstrap.cc:654] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:03.988665 5409 tablet_bootstrap.cc:492] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: No bootstrap required, opened a new log
I20260805 23:05:03.988762 5409 ts_tablet_manager.cc:1397] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:05:03.989084 5412 ts_tablet_manager.cc:1428] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260805 23:05:03.989444 5409 raft_consensus.cc:348] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.989568 5409 raft_consensus.cc:374] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.989612 5409 raft_consensus.cc:729] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2afcfa1adc4efebf29e69fc24326ad, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.989760 5409 consensus_queue.cc:260] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.990044 5410 tablet_bootstrap.cc:492] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: No bootstrap required, opened a new log
I20260805 23:05:03.990093 5409 ts_tablet_manager.cc:1428] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:03.990137 5410 ts_tablet_manager.cc:1397] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260805 23:05:03.990499 5419 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:03.990629 5410 raft_consensus.cc:348] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.990708 5419 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:03.990759 5410 raft_consensus.cc:374] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:03.990803 5410 raft_consensus.cc:729] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0140a7c7cf7d46e6aa8b352888c9b5bf, State: Initialized, Role: FOLLOWER
I20260805 23:05:03.990936 5410 consensus_queue.cc:260] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:03.990025 5417 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:03.991159 5417 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:03.991205 5410 ts_tablet_manager.cc:1428] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:03.991385 5431 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:03.991529 5431 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:04.134994 5431 raft_consensus.cc:482] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:04.135150 5431 raft_consensus.cc:504] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:04.135673 5431 leader_election.cc:288] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a2afcfa1adc4efebf29e69fc24326ad (127.3.12.65:46561), f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171)
I20260805 23:05:04.136243 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2333c324881e409f9a798e7dba89f04a" candidate_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f663865ada3045238fc5adc3d0936862" is_pre_election: true
I20260805 23:05:04.136480 5293 raft_consensus.cc:2454] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0140a7c7cf7d46e6aa8b352888c9b5bf in term 0.
I20260805 23:05:04.136421 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2333c324881e409f9a798e7dba89f04a" candidate_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" is_pre_election: true
I20260805 23:05:04.136643 4899 raft_consensus.cc:2454] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0140a7c7cf7d46e6aa8b352888c9b5bf in term 0.
I20260805 23:05:04.136878 5002 leader_election.cc:302] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0140a7c7cf7d46e6aa8b352888c9b5bf, f663865ada3045238fc5adc3d0936862; no voters:
I20260805 23:05:04.137082 5431 raft_consensus.cc:2781] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:04.137171 5431 raft_consensus.cc:482] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:04.137225 5431 raft_consensus.cc:3037] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:04.138594 5431 raft_consensus.cc:504] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:04.138976 5431 leader_election.cc:288] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [CANDIDATE]: Term 1 election: Requested vote from peers 3a2afcfa1adc4efebf29e69fc24326ad (127.3.12.65:46561), f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171)
I20260805 23:05:04.139309 4899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2333c324881e409f9a798e7dba89f04a" candidate_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2afcfa1adc4efebf29e69fc24326ad"
I20260805 23:05:04.139384 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2333c324881e409f9a798e7dba89f04a" candidate_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f663865ada3045238fc5adc3d0936862"
I20260805 23:05:04.139446 4899 raft_consensus.cc:3037] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:04.139487 5293 raft_consensus.cc:3037] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:04.141005 4899 raft_consensus.cc:2454] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0140a7c7cf7d46e6aa8b352888c9b5bf in term 1.
I20260805 23:05:04.141005 5293 raft_consensus.cc:2454] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0140a7c7cf7d46e6aa8b352888c9b5bf in term 1.
I20260805 23:05:04.141314 5002 leader_election.cc:302] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0140a7c7cf7d46e6aa8b352888c9b5bf, f663865ada3045238fc5adc3d0936862; no voters:
I20260805 23:05:04.141533 5431 raft_consensus.cc:2781] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:04.141634 5431 raft_consensus.cc:686] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 LEADER]: Becoming Leader. State: Replica: 0140a7c7cf7d46e6aa8b352888c9b5bf, State: Running, Role: LEADER
I20260805 23:05:04.141809 5431 consensus_queue.cc:237] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } }
I20260805 23:05:04.142570 5422 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: TxnStatusTablet state changed. Reason: New leader 0140a7c7cf7d46e6aa8b352888c9b5bf. Latest consensus state: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:04.142685 5422 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:04.143033 5467 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260805 23:05:04.143250 5467 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260805 23:05:04.143769 4677 catalog_manager.cc:5495] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0140a7c7cf7d46e6aa8b352888c9b5bf (127.3.12.66). New cstate: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:04.185921 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.182092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38663 (local address 127.0.0.1:38136)
0805 23:05:04.182294 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:04.182311 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:04.182427 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:04.182858 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:04.182860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:04.182868 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:04.182968 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:04.182972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.183969 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.183971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:04.184632 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:04.184657 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.185233 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.185236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:04.185303 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.185349 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:04.185666 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:04.185679 (+ 13us) client_negotiation.cc:769] Sending connection context
0805 23:05:04.185732 (+ 53us) client_negotiation.cc:240] Negotiation successful
0805 23:05:04.185804 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260805 23:05:04.185969 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.182238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.3.12.66:38663)
0805 23:05:04.182622 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:04.182627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:04.182640 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:05:04.182690 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:04.182693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:04.182743 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:04.182821 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:04.183083 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.183853 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.184766 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.185121 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.185227 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.185481 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:04.185565 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:04.185639 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:04.185768 (+ 129us) server_negotiation.cc:299] Negotiation successful
0805 23:05:04.185825 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260805 23:05:04.188953 4899 raft_consensus.cc:1261] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Refusing update from remote peer 0140a7c7cf7d46e6aa8b352888c9b5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:04.189121 5293 raft_consensus.cc:1261] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Refusing update from remote peer 0140a7c7cf7d46e6aa8b352888c9b5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:04.189476 5431 consensus_queue.cc:1035] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:04.189647 5422 consensus_queue.cc:1035] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:04.191165 5419 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: TxnStatusTablet state changed. Reason: New leader 0140a7c7cf7d46e6aa8b352888c9b5bf. Latest consensus state: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:04.191285 5419 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:04.192464 5431 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:04.192569 5431 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:04.192651 5425 mvcc.cc:205] Tried to move back new op lower bound from 7315337642752634880 to 7315337642566524928. Current Snapshot: MvccSnapshot[applied={T|T < 7315337642752634880}]
I20260805 23:05:04.193250 5417 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: TxnStatusTablet state changed. Reason: New leader 0140a7c7cf7d46e6aa8b352888c9b5bf. Latest consensus state: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:04.193358 5417 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:04.194758 5426 mvcc.cc:205] Tried to move back new op lower bound from 7315337642752634880 to 7315337642566524928. Current Snapshot: MvccSnapshot[applied={T|T < 7315337642752634880}]
I20260805 23:05:04.196148 5417 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:04.196274 5417 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P 3a2afcfa1adc4efebf29e69fc24326ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:04.199196 5418 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:04.199312 5418 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P 0140a7c7cf7d46e6aa8b352888c9b5bf: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:04.200558 5419 tablet_replica.cc:439] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2afcfa1adc4efebf29e69fc24326ad" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46561 } } peers { permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 } } peers { permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 } } }
I20260805 23:05:04.200673 5419 tablet_replica.cc:442] T 2333c324881e409f9a798e7dba89f04a P f663865ada3045238fc5adc3d0936862: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:04.203677 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.198478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.3.12.65:46561)
0805 23:05:04.199023 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:04.199050 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:04.199062 (+ 12us) server_negotiation.cc:407] Connection header received
0805 23:05:04.199160 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:04.199162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:04.199226 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:04.199400 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:04.199659 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.200645 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.201669 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.202119 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.202236 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.202527 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:04.203003 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:04.203139 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:04.203203 (+ 64us) server_negotiation.cc:299] Negotiation successful
0805 23:05:04.203261 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":185,"threads_started":1}
I20260805 23:05:04.216019 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.211467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:50579 (local address 127.3.12.66:38663)
0805 23:05:04.211583 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:04.211588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:04.212051 (+ 463us) server_negotiation.cc:407] Connection header received
0805 23:05:04.212217 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:04.212220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:04.212302 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:04.212385 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:04.212763 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.213542 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.214661 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.214999 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.215104 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.215473 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:04.215563 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:04.215637 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:04.215819 (+ 182us) server_negotiation.cc:299] Negotiation successful
0805 23:05:04.215875 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260805 23:05:04.216118 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.211444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38663 (local address 127.3.12.65:50579)
0805 23:05:04.211955 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:04.211969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:04.212114 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:04.212405 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:04.212407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:04.212414 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:04.212630 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:04.212636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.213688 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.213691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:04.214523 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:04.214534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.215115 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.215118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:04.215250 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.215302 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:04.215652 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:04.215658 (+ 6us) client_negotiation.cc:769] Sending connection context
0805 23:05:04.215743 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 23:05:04.215837 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":210,"threads_started":1}
I20260805 23:05:04.223726 5092 raft_consensus.cc:1261] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Refusing update from remote peer 3a2afcfa1adc4efebf29e69fc24326ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:04.224210 5293 raft_consensus.cc:1261] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Refusing update from remote peer 3a2afcfa1adc4efebf29e69fc24326ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:04.224474 5448 consensus_queue.cc:1035] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0140a7c7cf7d46e6aa8b352888c9b5bf" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:04.224663 5417 consensus_queue.cc:1035] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f663865ada3045238fc5adc3d0936862" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:04.248319 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.244429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:41339 (local address 127.3.12.65:46561)
0805 23:05:04.244557 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:04.244561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:04.244844 (+ 283us) server_negotiation.cc:407] Connection header received
0805 23:05:04.244936 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:04.244939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:04.244993 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:04.245081 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:04.245281 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.246015 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.247185 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.247434 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.247555 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.247843 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:04.247927 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:04.248009 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:04.248099 (+ 90us) server_negotiation.cc:299] Negotiation successful
0805 23:05:04.248153 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:05:04.248350 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.244179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:46561 (local address 127.3.12.66:41339)
0805 23:05:04.244748 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:04.244762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:04.244858 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:04.245082 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:04.245084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:04.245089 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:04.245185 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:04.245189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.246135 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.246155 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:04.246878 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:04.246890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.247555 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.247559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:04.247647 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.247703 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:04.248014 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:04.248023 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 23:05:04.248088 (+ 65us) client_negotiation.cc:240] Negotiation successful
0805 23:05:04.248183 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":180,"threads_started":1}
I20260805 23:05:04.279239 4676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:04.279497 4676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:04.282534 4676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fa746eed6584d6f8b4f85dac7771686: Sending DeleteTablet for 3 replicas of tablet b57104affe9947a985ac0447e1aed0ed
I20260805 23:05:04.283350 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet b57104affe9947a985ac0447e1aed0ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:04 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:47862
I20260805 23:05:04.283424 5262 tablet_service.cc:1514] Processing DeleteTablet for tablet b57104affe9947a985ac0447e1aed0ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:04 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:34134
I20260805 23:05:04.283532 5439 tablet_replica.cc:330] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: stopping tablet replica
I20260805 23:05:04.283650 5439 raft_consensus.cc:2229] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:04.283694 5438 tablet_replica.cc:330] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: stopping tablet replica
I20260805 23:05:04.283845 5438 raft_consensus.cc:2229] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:04.283942 5439 raft_consensus.cc:2258] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:04.284018 5438 raft_consensus.cc:2258] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:04.284997 5439 ts_tablet_manager.cc:1905] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:04.287685 5438 ts_tablet_manager.cc:1905] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:04.288364 5439 ts_tablet_manager.cc:1918] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:04.288442 5439 log.cc:1196] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/b57104affe9947a985ac0447e1aed0ed
I20260805 23:05:04.288748 5439 ts_tablet_manager.cc:1939] T b57104affe9947a985ac0447e1aed0ed P 3a2afcfa1adc4efebf29e69fc24326ad: Deleting consensus metadata
I20260805 23:05:04.289423 4655 catalog_manager.cc:4841] TS 3a2afcfa1adc4efebf29e69fc24326ad (127.3.12.65:46561): tablet b57104affe9947a985ac0447e1aed0ed (table test-table [id=fd3ed7241a994966a6cbd6ae1b3c07d2]) successfully deleted
I20260805 23:05:04.291102 5438 ts_tablet_manager.cc:1918] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:04.291189 5438 log.cc:1196] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/b57104affe9947a985ac0447e1aed0ed
I20260805 23:05:04.291460 5438 ts_tablet_manager.cc:1939] T b57104affe9947a985ac0447e1aed0ed P f663865ada3045238fc5adc3d0936862: Deleting consensus metadata
I20260805 23:05:04.292310 4657 catalog_manager.cc:4841] TS f663865ada3045238fc5adc3d0936862 (127.3.12.67:42171): tablet b57104affe9947a985ac0447e1aed0ed (table test-table [id=fd3ed7241a994966a6cbd6ae1b3c07d2]) successfully deleted
I20260805 23:05:04.292315 5066 tablet_service.cc:1514] Processing DeleteTablet for tablet b57104affe9947a985ac0447e1aed0ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:04 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:38132
I20260805 23:05:04.292727 5496 tablet_replica.cc:330] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: stopping tablet replica
I20260805 23:05:04.292886 5496 raft_consensus.cc:2229] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:04.293041 5496 raft_consensus.cc:2258] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:04.293890 5496 ts_tablet_manager.cc:1905] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:04.297049 5496 ts_tablet_manager.cc:1918] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:04.297154 5496 log.cc:1196] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/b57104affe9947a985ac0447e1aed0ed
I20260805 23:05:04.297505 5496 ts_tablet_manager.cc:1939] T b57104affe9947a985ac0447e1aed0ed P 0140a7c7cf7d46e6aa8b352888c9b5bf: Deleting consensus metadata
I20260805 23:05:04.298277 4655 catalog_manager.cc:4841] TS 0140a7c7cf7d46e6aa8b352888c9b5bf (127.3.12.66:38663): tablet b57104affe9947a985ac0447e1aed0ed (table test-table [id=fd3ed7241a994966a6cbd6ae1b3c07d2]) successfully deleted
Aug 05 23:05:04 dist-test-slave-2d19 krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:05:04.308845 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.286798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.3.12.65:46561)
0805 23:05:04.286950 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:04.286954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:04.292995 (+ 6041us) server_negotiation.cc:407] Connection header received
0805 23:05:04.293260 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:04.293263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:04.293315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:04.293369 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:04.294644 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.295580 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.297451 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.297848 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.297930 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.303286 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:04.303298 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:04.303317 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:04.303354 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:04.305979 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:04.306889 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:04.306918 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:04.306920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:04.307042 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:04.307442 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:04.307446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:04.307448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:04.307773 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:04.308012 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:04.308465 (+ 453us) server_negotiation.cc:299] Negotiation successful
0805 23:05:04.308607 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260805 23:05:04.310945 4957 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47878
I20260805 23:05:04.335502 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:04.324755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.3.12.126:41011)
0805 23:05:04.324930 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:04.324936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:04.325120 (+ 184us) server_negotiation.cc:407] Connection header received
0805 23:05:04.325329 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:04.325332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:04.325387 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:04.325453 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:04.326451 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.327365 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.328678 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:04.329056 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:04.329138 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:04.331035 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:04.331039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:04.331042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:04.331072 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:04.333119 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:04.333675 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:04.333679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:04.333682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:04.333837 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:04.334334 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:04.334338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:04.334340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:04.334644 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:04.334756 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:04.335173 (+ 417us) server_negotiation.cc:299] Negotiation successful
0805 23:05:04.335319 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260805 23:05:04.336536 4676 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35620
I20260805 23:05:04.338152 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 4779
I20260805 23:05:04.352877 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 4977
I20260805 23:05:04.370262 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 5166
I20260805 23:05:04.381940 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 4633
2026-08-05T23:05:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3508 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:04 dist-test-slave-2d19 krb5kdc[5505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:04 dist-test-slave-2d19 krb5kdc[5505](info): set up 2 sockets
Aug 05 23:05:04 dist-test-slave-2d19 krb5kdc[5505](info): commencing operation
krb5kdc: starting...
W20260805 23:05:05.631457 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:05 dist-test-slave-2d19 krb5kdc[5505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:05.769876 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:34095
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:05.885571 5528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:05.885883 5528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:05.885957 5528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:05.890009 5528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:05.890100 5528 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260805 23:05:05.890131 5528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:05.890174 5528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:05.890213 5528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:05.890249 5528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:05.895660 5528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:34095
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:05.896951 5528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:05.898075 5528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:05:05.907341 5528 server_base.cc:1034] running on GCE node
W20260805 23:05:05.907274 5540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:05.907236 5534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:05.907250 5537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:05.908072 5528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:05.909268 5528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:05.910511 5528 hybrid_clock.cc:648] HybridClock initialized: now 1785971105910488 us; error 52 us; skew 500 ppm
Aug 05 23:05:05 dist-test-slave-2d19 krb5kdc[5505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:05.920307 5528 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:05.921368 5528 webserver.cc:460] Webserver started at http://127.3.12.126:43623/ using document root <none> and password file <none>
I20260805 23:05:05.921995 5528 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:05.922063 5528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:05.922320 5528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:05.924599 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "16c25aca341845d4bc8aa12d08c1a98c"
format_stamp: "Formatted at 2026-08-05 23:05:05 on dist-test-slave-2d19"
server_key: "872bb89acd8d9b8fa82b013b49f4a6a3"
server_key_iv: "9df657e4c13aff2451de52ff461bc734"
server_key_version: "encryptionkey@0"
I20260805 23:05:05.925207 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "16c25aca341845d4bc8aa12d08c1a98c"
format_stamp: "Formatted at 2026-08-05 23:05:05 on dist-test-slave-2d19"
server_key: "872bb89acd8d9b8fa82b013b49f4a6a3"
server_key_iv: "9df657e4c13aff2451de52ff461bc734"
server_key_version: "encryptionkey@0"
I20260805 23:05:05.929277 5528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:05.931891 5552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:05.933295 5528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:05:05.933451 5528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "16c25aca341845d4bc8aa12d08c1a98c"
format_stamp: "Formatted at 2026-08-05 23:05:05 on dist-test-slave-2d19"
server_key: "872bb89acd8d9b8fa82b013b49f4a6a3"
server_key_iv: "9df657e4c13aff2451de52ff461bc734"
server_key_version: "encryptionkey@0"
I20260805 23:05:05.933580 5528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:05.950985 5528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:05.954738 5528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:05.954967 5528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:05.963261 5528 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:34095
I20260805 23:05:05.963289 5643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:34095 every 8 connection(s)
I20260805 23:05:05.964366 5528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:05.967224 5644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:05.968554 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 5528
I20260805 23:05:05.968683 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:05.968981 3121 external_mini_cluster.cc:1428] Setting key ad0192b0e7a7b1a582012b1163de8c89
I20260805 23:05:05.973729 5644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: Bootstrap starting.
I20260805 23:05:05.976063 5644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:05.976891 5644 log.cc:824] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:05:05 dist-test-slave-2d19 krb5kdc[5505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:05.979048 5644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: No bootstrap required, opened a new log
I20260805 23:05:05.982744 5644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25aca341845d4bc8aa12d08c1a98c" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34095 } }
I20260805 23:05:05.982993 5644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:05.983070 5644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c25aca341845d4bc8aa12d08c1a98c, State: Initialized, Role: FOLLOWER
I20260805 23:05:05.983572 5644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25aca341845d4bc8aa12d08c1a98c" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34095 } }
I20260805 23:05:05.983737 5644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:05.983827 5644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:05.983992 5644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:05.985062 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:05.970617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.3.12.126:34095)
0805 23:05:05.971108 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:05.971124 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:05.971174 (+ 50us) server_negotiation.cc:407] Connection header received
0805 23:05:05.971859 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:05.971877 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:05.972242 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:05.972684 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:05.973522 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:05.974591 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:05.975489 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:05.975989 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:05.976174 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:05.979007 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:05.979012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:05.979026 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:05.979055 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:05.981625 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:05.982152 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:05.982157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:05.982162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:05.982282 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:05.982629 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:05.982632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:05.982634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:05.983114 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:05.983377 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:05.983674 (+ 297us) server_negotiation.cc:299] Negotiation successful
0805 23:05:05.984230 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":145,"threads_started":1}
I20260805 23:05:05.985044 5644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25aca341845d4bc8aa12d08c1a98c" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34095 } }
I20260805 23:05:05.985430 5644 leader_election.cc:302] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16c25aca341845d4bc8aa12d08c1a98c; no voters:
I20260805 23:05:05.985803 5644 leader_election.cc:288] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:05.985883 5653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:05.986085 5653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [term 1 LEADER]: Becoming Leader. State: Replica: 16c25aca341845d4bc8aa12d08c1a98c, State: Running, Role: LEADER
I20260805 23:05:05.987500 5653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25aca341845d4bc8aa12d08c1a98c" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34095 } }
I20260805 23:05:05.987154 5644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:05.989450 5654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16c25aca341845d4bc8aa12d08c1a98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25aca341845d4bc8aa12d08c1a98c" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34095 } } }
I20260805 23:05:05.989593 5654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:05.990027 5668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:05.989952 5656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16c25aca341845d4bc8aa12d08c1a98c. Latest consensus state: current_term: 1 leader_uuid: "16c25aca341845d4bc8aa12d08c1a98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25aca341845d4bc8aa12d08c1a98c" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34095 } } }
I20260805 23:05:05.990721 5656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:05.993451 5668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:05.999432 5668 catalog_manager.cc:1347] Generated new cluster ID: 6328c59e45d24fbab70ee1492186ba9c
I20260805 23:05:05.999521 5668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:06.031704 5668 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:06.032791 5668 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:06.046278 5668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: Generated new TSK 0
I20260805 23:05:06.047084 5668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:05:06.088390 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34095
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:06.222250 5684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:06.222543 5684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:06.222591 5684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:06.226655 5684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:06.226717 5684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:06.226758 5684 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260805 23:05:06.226773 5684 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:06.226809 5684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:06.232100 5684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:34095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:06.233217 5684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:06.234230 5684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:06.244391 5694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:06.244477 5684 server_base.cc:1034] running on GCE node
W20260805 23:05:06.244385 5693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:06.244385 5696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:06.245065 5684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:06.245752 5684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:06.246948 5684 hybrid_clock.cc:648] HybridClock initialized: now 1785971106246916 us; error 59 us; skew 500 ppm
Aug 05 23:05:06 dist-test-slave-2d19 krb5kdc[5505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971106, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:06.249918 5684 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:05:06.250838 5684 webserver.cc:460] Webserver started at http://127.3.12.65:34585/ using document root <none> and password file <none>
I20260805 23:05:06.251443 5684 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:06.251546 5684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:06.251789 5684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:06.253964 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "b1dc6c17624d4369ad0fad1373a1bea3"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "05111774a7b29c86547fbb9f9dd54961"
server_key_iv: "d42f30a1273a0f343f473537b6badd65"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.254558 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "b1dc6c17624d4369ad0fad1373a1bea3"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "05111774a7b29c86547fbb9f9dd54961"
server_key_iv: "d42f30a1273a0f343f473537b6badd65"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.258487 5684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260805 23:05:06.260943 5706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:06.262166 5684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:06.262308 5684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "b1dc6c17624d4369ad0fad1373a1bea3"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "05111774a7b29c86547fbb9f9dd54961"
server_key_iv: "d42f30a1273a0f343f473537b6badd65"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.262421 5684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:06.281960 5684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:06.285319 5684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:06.285537 5684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:06.287181 5684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:06.287258 5684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:06.287333 5684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:06.287385 5684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:06 dist-test-slave-2d19 krb5kdc[5505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971106, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:06.298977 5684 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:45319
I20260805 23:05:06.300110 5684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:06.300390 5882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:45319 every 8 connection(s)
I20260805 23:05:06.302846 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.288281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:47563 (local address 127.3.12.126:34095)
0805 23:05:06.288402 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.288406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.289586 (+ 1180us) server_negotiation.cc:407] Connection header received
0805 23:05:06.290868 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.290872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.290944 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.291054 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:06.292726 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.293402 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.295311 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.295717 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.295818 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.299000 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:06.299004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:06.299007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:06.299043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:06.300642 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.301251 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.301255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.301257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.301353 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.301722 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.301726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.301727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.301920 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:06.302041 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.302526 (+ 485us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.302670 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260805 23:05:06.303707 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.288854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34095 (local address 127.3.12.65:47563)
0805 23:05:06.289435 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.289481 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.290600 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.291377 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.291388 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.291841 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:06.292561 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.292577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.293553 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.293560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.295166 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.295182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.295889 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.295896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.296139 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.296874 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:06.296901 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:06.298816 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:06.300784 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.300792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.300806 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.301134 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.301451 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.301455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.301457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.301607 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.302033 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:06.302039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:06.302284 (+ 245us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.302464 (+ 180us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.302788 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":145,"threads_started":1}
I20260805 23:05:06.305094 5886 heartbeater.cc:347] Connected to a master server at 127.3.12.126:34095
I20260805 23:05:06.305254 5886 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:06.305756 5886 heartbeater.cc:511] Master 127.3.12.126:34095 requested a full tablet report, sending...
I20260805 23:05:06.307255 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 5684
I20260805 23:05:06.307386 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:06.307456 5575 ts_manager.cc:194] Registered new tserver with Master: b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319)
I20260805 23:05:06.307760 3121 external_mini_cluster.cc:1428] Setting key 2f3b3d5e8d98b6ac7e5591b5b7ff634b
I20260805 23:05:06.309036 5575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:47563
I20260805 23:05:06.315209 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.307701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:38951 (local address 127.3.12.126:34095)
0805 23:05:06.307804 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.307808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.307822 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:06.307998 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.308001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.308042 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.308125 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:06.308685 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.309243 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.310297 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.310612 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.310701 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.312006 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:06.312010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:06.312013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:06.312037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:06.313548 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.314006 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.314011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.314013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.314095 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.314381 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.314385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.314387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.314564 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:06.314674 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.314938 (+ 264us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.315051 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260805 23:05:06.315220 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.307371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34095 (local address 127.3.12.65:38951)
0805 23:05:06.307730 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.307746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.307890 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.308168 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.308171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.308436 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:06.308568 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.308576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.309378 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.309382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.310150 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.310161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.310849 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.310853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.310942 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.311497 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:06.311511 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:06.311855 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:06.313665 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.313671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.313673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.313901 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.314193 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.314196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.314198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.314287 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.314664 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:06.314668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:06.314766 (+ 98us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.314874 (+ 108us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.315014 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:06.339301 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34095
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:06.460073 5897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:06.460397 5897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:06.460489 5897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:06.468194 5897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:06.468317 5897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:06.468413 5897 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260805 23:05:06.468451 5897 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:06.468518 5897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:06.478761 5897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:34095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:06.480368 5897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:06.481921 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:06.495829 5910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:06.495990 5897 server_base.cc:1034] running on GCE node
W20260805 23:05:06.495837 5912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:06.495827 5915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:06.496929 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:06.498035 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:06.499272 5897 hybrid_clock.cc:648] HybridClock initialized: now 1785971106499235 us; error 52 us; skew 500 ppm
Aug 05 23:05:06 dist-test-slave-2d19 krb5kdc[5505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971106, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:06.503753 5897 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:06.505052 5897 webserver.cc:460] Webserver started at http://127.3.12.66:36553/ using document root <none> and password file <none>
I20260805 23:05:06.505759 5897 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:06.505858 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:06.506115 5897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:06.508455 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "8aef2324cfdb4b2288b8798993eb5030"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "f87a1844af333cead4d55bac07db4929"
server_key_iv: "7b8eab0aa492c284bce9df2de86844e5"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.509053 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "8aef2324cfdb4b2288b8798993eb5030"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "f87a1844af333cead4d55bac07db4929"
server_key_iv: "7b8eab0aa492c284bce9df2de86844e5"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.513162 5897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260805 23:05:06.515748 5932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:06.517266 5897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:05:06.517437 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "8aef2324cfdb4b2288b8798993eb5030"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "f87a1844af333cead4d55bac07db4929"
server_key_iv: "7b8eab0aa492c284bce9df2de86844e5"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.517582 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:06.531055 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:06.534688 5897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:06.534910 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:06.536518 5897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:06.536597 5897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:06.536672 5897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:06.536715 5897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:06 dist-test-slave-2d19 krb5kdc[5505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971106, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:06.548578 5897 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:35041
I20260805 23:05:06.548650 6108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:35041 every 8 connection(s)
I20260805 23:05:06.549540 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:06.552536 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.537451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:39721 (local address 127.3.12.126:34095)
0805 23:05:06.537586 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.537589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.538787 (+ 1198us) server_negotiation.cc:407] Connection header received
0805 23:05:06.539806 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.539810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.539869 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.539929 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:06.541594 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.542517 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.544365 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.544732 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.544856 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.548403 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:06.548408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:06.548412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:06.548447 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:06.550270 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.550823 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.550826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.550828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.550919 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.551350 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.551353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.551354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.551585 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:06.551668 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.552187 (+ 519us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.552382 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:05:06.553411 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.538082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34095 (local address 127.3.12.66:39721)
0805 23:05:06.538613 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.538681 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.539524 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.540449 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.540456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.540797 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:06.541403 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.541443 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.542662 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.542665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.544203 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.544218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.544974 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.544979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.545216 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.545926 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:06.546040 (+ 114us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:06.548162 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:06.550425 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.550431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.550453 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.550717 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.551022 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.551025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.551027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.551224 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.551684 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:06.551689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:06.551946 (+ 257us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.552098 (+ 152us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.552446 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":149,"threads_started":1}
I20260805 23:05:06.554734 6109 heartbeater.cc:347] Connected to a master server at 127.3.12.126:34095
I20260805 23:05:06.554908 6109 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:06.555459 6109 heartbeater.cc:511] Master 127.3.12.126:34095 requested a full tablet report, sending...
I20260805 23:05:06.556594 5575 ts_manager.cc:194] Registered new tserver with Master: 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041)
I20260805 23:05:06.557312 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 5897
I20260805 23:05:06.557396 5575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:39721
I20260805 23:05:06.557441 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:06.557793 3121 external_mini_cluster.cc:1428] Setting key d250326e851916c0feff71862df16303
I20260805 23:05:06.565733 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.557747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:33285 (local address 127.3.12.126:34095)
0805 23:05:06.557857 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.557862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.557877 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:06.557912 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.557914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.557957 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.558046 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:06.558613 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.559337 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.560730 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.560946 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.561032 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.562347 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:06.562352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:06.562355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:06.562383 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:06.563846 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.564429 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.564431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.564433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.564510 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.564845 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.564847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.564848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.565053 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:06.565137 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.565462 (+ 325us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.565589 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:05:06.565948 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.557159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34095 (local address 127.3.12.66:33285)
0805 23:05:06.557517 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.557532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.557624 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.558083 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.558086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.558357 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:06.558480 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.558486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.559493 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.559498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.560553 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.560569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.561067 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.561074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.561224 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.561845 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:06.561857 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:06.562197 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:06.563969 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.563975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.563981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.564327 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.564614 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.564618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.564622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.564737 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.565165 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:06.565169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:06.565283 (+ 114us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.565410 (+ 127us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.565591 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:06.592514 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34095
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:06.706698 6124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:06.706956 6124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:06.707033 6124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:06.711301 6124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:06.711373 6124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:06.711416 6124 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260805 23:05:06.711470 6124 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:06.711535 6124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:06.716881 6124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:34095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:06.717969 6124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:06.719034 6124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:06.729374 6133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:06.729403 6132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:06.729554 6124 server_base.cc:1034] running on GCE node
W20260805 23:05:06.729722 6135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:06.730158 6124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:06.730839 6124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:06.732040 6124 hybrid_clock.cc:648] HybridClock initialized: now 1785971106732020 us; error 37 us; skew 500 ppm
Aug 05 23:05:06 dist-test-slave-2d19 krb5kdc[5505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971106, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:06.734891 6124 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:06.735790 6124 webserver.cc:460] Webserver started at http://127.3.12.67:34891/ using document root <none> and password file <none>
I20260805 23:05:06.736413 6124 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:06.736483 6124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:06.736670 6124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:06.738960 6124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "0bbc802857f34ca794d76681f52dd582"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "5bf7bd13abd488a076f155fb60920a81"
server_key_iv: "57c3742549729f0cee4928a777907b2b"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.739508 6124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "0bbc802857f34ca794d76681f52dd582"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "5bf7bd13abd488a076f155fb60920a81"
server_key_iv: "57c3742549729f0cee4928a777907b2b"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.743371 6124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:05:06.745880 6145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:06.747159 6124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:06.747263 6124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "0bbc802857f34ca794d76681f52dd582"
format_stamp: "Formatted at 2026-08-05 23:05:06 on dist-test-slave-2d19"
server_key: "5bf7bd13abd488a076f155fb60920a81"
server_key_iv: "57c3742549729f0cee4928a777907b2b"
server_key_version: "encryptionkey@0"
I20260805 23:05:06.747354 6124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:06.775061 6124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:06.778111 6124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:06.778272 6124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:06.779753 6124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:06.779806 6124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:06.779871 6124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:06.779906 6124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:06 dist-test-slave-2d19 krb5kdc[5505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971106, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:06.791347 6124 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:43535
I20260805 23:05:06.791402 6316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:43535 every 8 connection(s)
I20260805 23:05:06.792359 6124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:06.794281 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.780676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:59999 (local address 127.3.12.126:34095)
0805 23:05:06.780815 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.780819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.781787 (+ 968us) server_negotiation.cc:407] Connection header received
0805 23:05:06.783151 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.783157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.783222 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.783346 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:06.784846 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.785465 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.786659 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.787068 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.787201 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.790371 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:06.790375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:06.790380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:06.790411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:06.792121 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.792754 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.792758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.792760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.792862 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.793261 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.793264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.793266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.793499 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:06.793591 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.794058 (+ 467us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.794161 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260805 23:05:06.795055 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.781072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34095 (local address 127.3.12.67:59999)
0805 23:05:06.781598 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.781661 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.782852 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.783576 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.783583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.784025 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:06.784695 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.784710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.785595 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.785598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.786519 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.786533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.787278 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.787285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.787504 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.788150 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:06.788170 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:06.790175 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:06.792245 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.792276 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.792295 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.792625 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.792980 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.792984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.792985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.793147 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.793606 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:06.793611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:06.793838 (+ 227us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.793985 (+ 147us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.794232 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260805 23:05:06.796337 6318 heartbeater.cc:347] Connected to a master server at 127.3.12.126:34095
I20260805 23:05:06.796481 6318 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:06.797047 6318 heartbeater.cc:511] Master 127.3.12.126:34095 requested a full tablet report, sending...
I20260805 23:05:06.798151 5577 ts_manager.cc:194] Registered new tserver with Master: 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535)
I20260805 23:05:06.798965 5577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:59999
I20260805 23:05:06.800143 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 6124
I20260805 23:05:06.800287 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:06.800652 3121 external_mini_cluster.cc:1428] Setting key 71dd973981fea28a5cdb7fd14ab820ab
I20260805 23:05:06.802659 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:05:06.806988 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.799041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34095 (local address 127.3.12.67:50041)
0805 23:05:06.799554 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.799567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.799695 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.799985 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.799988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.800206 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:06.800352 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.800361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.801378 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.801380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.802044 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.802054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.802521 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.802525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.802622 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.803153 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:06.803166 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:06.803482 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:06.805535 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.805538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.805539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.805734 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.806040 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:06.806042 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:06.806044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:06.806119 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:06.806574 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:06.806577 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:06.806653 (+ 76us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.806738 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.806844 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":116,"threads_started":1}
I20260805 23:05:06.807192 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.798802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:50041 (local address 127.3.12.126:34095)
0805 23:05:06.798937 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.798943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.799651 (+ 708us) server_negotiation.cc:407] Connection header received
0805 23:05:06.799788 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.799791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.799855 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.799938 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:06.800482 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.801183 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.802173 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.802404 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.802515 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.803623 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:06.803626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:06.803629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:06.803653 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:06.805412 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.805836 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.805840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.805845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.805940 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.806220 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.806225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.806228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.806464 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:06.806578 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.806828 (+ 250us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.806962 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 05 23:05:06 dist-test-slave-2d19 krb5kdc[5505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 23:05:06 dist-test-slave-2d19 krb5kdc[5505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:06.831352 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.822983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.3.12.126:34095)
0805 23:05:06.823123 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.823127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.823139 (+ 12us) server_negotiation.cc:407] Connection header received
0805 23:05:06.823183 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.823185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.823224 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.823318 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:06.824057 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.824663 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.825569 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.825795 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.825904 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.828325 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:06.828329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:06.828331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:06.828357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:06.829589 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.830040 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.830044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.830046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.830130 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:06.830441 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:06.830445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:06.830447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:06.830649 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:06.830798 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.831044 (+ 246us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.831167 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:05:06.834542 5577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:06.837721 5577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:06.852943 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.846173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:45319 (local address 127.0.0.1:57264)
0805 23:05:06.846767 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.846821 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.846958 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.848132 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.848135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.848155 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:06.848358 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.848365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.849745 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.849749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.851097 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.851109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.852315 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.852318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.852446 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.852494 (+ 48us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.852625 (+ 131us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.852753 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":150,"thread_start_us":91,"threads_started":1}
I20260805 23:05:06.852933 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.846301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.3.12.65:45319)
0805 23:05:06.847646 (+ 1345us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.847651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.847672 (+ 21us) server_negotiation.cc:407] Connection header received
0805 23:05:06.847763 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.847768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.847956 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.848393 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:06.848534 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.849614 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.851205 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.852070 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.852342 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.852510 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.852669 (+ 159us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.852789 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":87,"threads_started":1}
I20260805 23:05:06.853102 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.847040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.3.12.66:35041)
0805 23:05:06.847480 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.847493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.847780 (+ 287us) server_negotiation.cc:407] Connection header received
0805 23:05:06.847855 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.847859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.848006 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.848153 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:06.848584 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.849327 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.851271 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.852122 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.852247 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.852455 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.852836 (+ 381us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.852931 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":122,"threads_started":1}
I20260805 23:05:06.852977 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.846245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43535 (local address 127.0.0.1:54384)
0805 23:05:06.847295 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.847309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.847396 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.848016 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.848022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.848044 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:06.848206 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.848214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.849750 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.849752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.851296 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.851305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.852579 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.852583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.852687 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.852710 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.852783 (+ 73us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.852849 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":272,"thread_start_us":104,"threads_started":1}
I20260805 23:05:06.853368 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.846497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:35041 (local address 127.0.0.1:40660)
0805 23:05:06.847634 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.847660 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.847758 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.848202 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.848205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.848221 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:06.848473 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.848476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.849449 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.849453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.851130 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.851148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.852517 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.852520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.852619 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.852641 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.852702 (+ 61us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.852752 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":119,"thread_start_us":95,"threads_started":1}
I20260805 23:05:06.853801 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.847036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.3.12.67:43535)
0805 23:05:06.847480 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.847493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.847521 (+ 28us) server_negotiation.cc:407] Connection header received
0805 23:05:06.847607 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.847613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.847858 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.848028 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:06.848347 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.849596 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.851416 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.852247 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.852369 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.852513 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.853559 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.853660 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":121,"threads_started":1}
I20260805 23:05:06.856195 5788 tablet_service.cc:1467] Processing CreateTablet for tablet 9e989df8c9364bcd997478d83f138dec (DEFAULT_TABLE table=test-table [id=3d5b3f1d800f418a84a2320f31d0044a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:06.856243 5996 tablet_service.cc:1467] Processing CreateTablet for tablet 9e989df8c9364bcd997478d83f138dec (DEFAULT_TABLE table=test-table [id=3d5b3f1d800f418a84a2320f31d0044a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:06.856243 6217 tablet_service.cc:1467] Processing CreateTablet for tablet 9e989df8c9364bcd997478d83f138dec (DEFAULT_TABLE table=test-table [id=3d5b3f1d800f418a84a2320f31d0044a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:06.857343 6217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e989df8c9364bcd997478d83f138dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:06.857339 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e989df8c9364bcd997478d83f138dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:06.857344 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e989df8c9364bcd997478d83f138dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:06.863517 6353 tablet_bootstrap.cc:492] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Bootstrap starting.
I20260805 23:05:06.865141 6354 tablet_bootstrap.cc:492] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Bootstrap starting.
I20260805 23:05:06.866094 6353 tablet_bootstrap.cc:654] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:06.866093 6355 tablet_bootstrap.cc:492] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Bootstrap starting.
I20260805 23:05:06.867004 6353 log.cc:824] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:06.867393 6354 tablet_bootstrap.cc:654] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:06.868242 6355 tablet_bootstrap.cc:654] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:06.868306 6354 log.cc:824] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:06.869141 6355 log.cc:824] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:06.869243 6353 tablet_bootstrap.cc:492] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: No bootstrap required, opened a new log
I20260805 23:05:06.869441 6353 ts_tablet_manager.cc:1397] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:05:06.870406 6354 tablet_bootstrap.cc:492] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: No bootstrap required, opened a new log
I20260805 23:05:06.870602 6354 ts_tablet_manager.cc:1397] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:05:06.870949 6355 tablet_bootstrap.cc:492] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: No bootstrap required, opened a new log
I20260805 23:05:06.871146 6355 ts_tablet_manager.cc:1397] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 23:05:06.874039 6355 raft_consensus.cc:348] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:06.873935 6353 raft_consensus.cc:348] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:06.874282 6355 raft_consensus.cc:374] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:06.874284 6353 raft_consensus.cc:374] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:06.874338 6355 raft_consensus.cc:729] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbc802857f34ca794d76681f52dd582, State: Initialized, Role: FOLLOWER
I20260805 23:05:06.874341 6353 raft_consensus.cc:729] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dc6c17624d4369ad0fad1373a1bea3, State: Initialized, Role: FOLLOWER
I20260805 23:05:06.874502 6354 raft_consensus.cc:348] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:06.874748 6354 raft_consensus.cc:374] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:06.874794 6354 raft_consensus.cc:729] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aef2324cfdb4b2288b8798993eb5030, State: Initialized, Role: FOLLOWER
I20260805 23:05:06.874822 6353 consensus_queue.cc:260] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:06.874831 6355 consensus_queue.cc:260] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:06.875226 6354 consensus_queue.cc:260] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:06.875893 6355 ts_tablet_manager.cc:1428] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:05:06.875887 6353 ts_tablet_manager.cc:1428] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 23:05:06.876093 6354 ts_tablet_manager.cc:1428] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260805 23:05:06.876313 5886 heartbeater.cc:503] Master 127.3.12.126:34095 was elected leader, sending a full tablet report...
I20260805 23:05:06.876317 6318 heartbeater.cc:503] Master 127.3.12.126:34095 was elected leader, sending a full tablet report...
I20260805 23:05:06.876421 6109 heartbeater.cc:503] Master 127.3.12.126:34095 was elected leader, sending a full tablet report...
I20260805 23:05:06.983516 6359 raft_consensus.cc:482] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:06.983827 6359 raft_consensus.cc:504] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:06.985483 6359 leader_election.cc:288] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535), 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041)
I20260805 23:05:06.991843 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.986008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:49129 (local address 127.3.12.66:35041)
0805 23:05:06.986161 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.986168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.986590 (+ 422us) server_negotiation.cc:407] Connection header received
0805 23:05:06.986656 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.986659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.986718 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.986801 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:06.987380 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.988286 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.990275 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.991031 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.991175 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.991254 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.991602 (+ 348us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.991691 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 23:05:06.991844 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.985950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:58687 (local address 127.3.12.67:43535)
0805 23:05:06.986095 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:06.986099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:06.986351 (+ 252us) server_negotiation.cc:407] Connection header received
0805 23:05:06.986606 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:06.986611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:06.986680 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:06.986781 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:06.987266 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.988290 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.990275 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.991071 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.991192 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.991254 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:06.991602 (+ 348us) server_negotiation.cc:299] Negotiation successful
0805 23:05:06.991691 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:05:06.991914 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.985893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:35041 (local address 127.3.12.65:49129)
0805 23:05:06.986361 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.986375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.986507 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.986832 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.986836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.986962 (+ 126us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:06.987236 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.987248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.988429 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.988433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.990111 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.990126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.991198 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.991201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.991398 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.991469 (+ 71us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.991556 (+ 87us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.991670 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":206,"spinlock_wait_cycles":30848,"thread_start_us":116,"threads_started":1}
I20260805 23:05:06.991916 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:06.985731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43535 (local address 127.3.12.65:58687)
0805 23:05:06.986210 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:06.986226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:06.986383 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:06.986832 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:06.986836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:06.986883 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:06.987036 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.987069 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.988429 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.988433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.990116 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:06.990126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:06.991198 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:06.991201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:06.991378 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:06.991473 (+ 95us) client_negotiation.cc:769] Sending connection context
0805 23:05:06.991556 (+ 83us) client_negotiation.cc:240] Negotiation successful
0805 23:05:06.991670 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":184,"spinlock_wait_cycles":7808,"thread_start_us":173,"threads_started":1}
I20260805 23:05:06.992777 6034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e989df8c9364bcd997478d83f138dec" candidate_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aef2324cfdb4b2288b8798993eb5030" is_pre_election: true
I20260805 23:05:06.992825 6255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e989df8c9364bcd997478d83f138dec" candidate_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbc802857f34ca794d76681f52dd582" is_pre_election: true
I20260805 23:05:06.993134 6255 raft_consensus.cc:2454] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1dc6c17624d4369ad0fad1373a1bea3 in term 0.
I20260805 23:05:06.993147 6034 raft_consensus.cc:2454] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1dc6c17624d4369ad0fad1373a1bea3 in term 0.
I20260805 23:05:06.993659 5710 leader_election.cc:302] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbc802857f34ca794d76681f52dd582, b1dc6c17624d4369ad0fad1373a1bea3; no voters:
I20260805 23:05:06.993969 6359 raft_consensus.cc:2781] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:06.994066 6359 raft_consensus.cc:482] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:06.994110 6359 raft_consensus.cc:3037] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:06.995213 6359 raft_consensus.cc:504] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:06.995600 6359 leader_election.cc:288] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [CANDIDATE]: Term 1 election: Requested vote from peers 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535), 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041)
I20260805 23:05:06.996178 6034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e989df8c9364bcd997478d83f138dec" candidate_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aef2324cfdb4b2288b8798993eb5030"
I20260805 23:05:06.996311 6255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e989df8c9364bcd997478d83f138dec" candidate_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbc802857f34ca794d76681f52dd582"
I20260805 23:05:06.996368 6034 raft_consensus.cc:3037] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:06.996451 6255 raft_consensus.cc:3037] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:06.998450 6034 raft_consensus.cc:2454] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1dc6c17624d4369ad0fad1373a1bea3 in term 1.
I20260805 23:05:06.998450 6255 raft_consensus.cc:2454] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1dc6c17624d4369ad0fad1373a1bea3 in term 1.
I20260805 23:05:06.998823 5710 leader_election.cc:302] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbc802857f34ca794d76681f52dd582, b1dc6c17624d4369ad0fad1373a1bea3; no voters:
I20260805 23:05:06.999025 6359 raft_consensus.cc:2781] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:06.999282 6359 raft_consensus.cc:686] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 LEADER]: Becoming Leader. State: Replica: b1dc6c17624d4369ad0fad1373a1bea3, State: Running, Role: LEADER
I20260805 23:05:06.999660 6359 consensus_queue.cc:237] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } }
I20260805 23:05:07.003206 5577 catalog_manager.cc:5495] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65). New cstate: current_term: 1 leader_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } health_report { overall_health: UNKNOWN } } }
W20260805 23:05:07.043431 6319 tablet.cc:2369] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 23:05:07.051010 6113 tablet.cc:2369] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 23:05:07.051579 5887 tablet.cc:2369] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 05 23:05:07 dist-test-slave-2d19 krb5kdc[5505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:07.070919 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.062189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.3.12.126:34095)
0805 23:05:07.062333 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.062337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.062563 (+ 226us) server_negotiation.cc:407] Connection header received
0805 23:05:07.062682 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.062684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.062734 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.062808 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:07.063472 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.064007 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.064854 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.065083 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.065165 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.067843 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:07.067846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:07.067848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:07.067871 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:07.069329 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:07.069790 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:07.069793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:07.069794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:07.069870 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:07.070118 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:07.070120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:07.070121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:07.070304 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:07.070385 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.070691 (+ 306us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.070790 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:05:07.070920 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.062115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34095 (local address 127.0.0.1:46352)
0805 23:05:07.062464 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:07.062477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:07.062592 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:07.062854 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:07.062857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:07.063263 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:07.063362 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.063368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.064137 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.064139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.064752 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.064761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.065189 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.065192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.065263 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.065807 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:07.065819 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:07.067683 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:07.069442 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:07.069450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:07.069455 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:07.069698 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:07.069954 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:07.069957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:07.069958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:07.070037 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:07.070406 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:07.070412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:07.070521 (+ 109us) client_negotiation.cc:769] Sending connection context
0805 23:05:07.070617 (+ 96us) client_negotiation.cc:240] Negotiation successful
0805 23:05:07.070740 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":141,"threads_started":1}
I20260805 23:05:07.075338 5577 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:46352:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260805 23:05:07.076025 5577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:07.080869 5996 tablet_service.cc:1467] Processing CreateTablet for tablet 615015a3dddd4fc68c1fffc5025854a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf9ea7e6e61040b38a1bc43d55181d80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:07.080888 5788 tablet_service.cc:1467] Processing CreateTablet for tablet 615015a3dddd4fc68c1fffc5025854a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf9ea7e6e61040b38a1bc43d55181d80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:07.081116 6217 tablet_service.cc:1467] Processing CreateTablet for tablet 615015a3dddd4fc68c1fffc5025854a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cf9ea7e6e61040b38a1bc43d55181d80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:07.081684 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615015a3dddd4fc68c1fffc5025854a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:07.081684 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615015a3dddd4fc68c1fffc5025854a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:07.081790 6217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615015a3dddd4fc68c1fffc5025854a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:07.084658 6355 tablet_bootstrap.cc:492] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: Bootstrap starting.
I20260805 23:05:07.084767 6354 tablet_bootstrap.cc:492] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: Bootstrap starting.
I20260805 23:05:07.085705 6355 tablet_bootstrap.cc:654] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:07.085705 6354 tablet_bootstrap.cc:654] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:07.086628 6353 tablet_bootstrap.cc:492] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: Bootstrap starting.
I20260805 23:05:07.087531 6355 tablet_bootstrap.cc:492] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: No bootstrap required, opened a new log
I20260805 23:05:07.087642 6355 ts_tablet_manager.cc:1397] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:07.087936 6354 tablet_bootstrap.cc:492] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: No bootstrap required, opened a new log
I20260805 23:05:07.088030 6354 ts_tablet_manager.cc:1397] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:07.088120 6353 tablet_bootstrap.cc:654] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:07.088217 6355 raft_consensus.cc:348] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.088376 6355 raft_consensus.cc:374] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:07.088421 6355 raft_consensus.cc:729] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbc802857f34ca794d76681f52dd582, State: Initialized, Role: FOLLOWER
I20260805 23:05:07.088551 6355 consensus_queue.cc:260] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.088549 6354 raft_consensus.cc:348] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.088653 6354 raft_consensus.cc:374] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:07.088689 6354 raft_consensus.cc:729] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aef2324cfdb4b2288b8798993eb5030, State: Initialized, Role: FOLLOWER
I20260805 23:05:07.088845 6355 ts_tablet_manager.cc:1428] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260805 23:05:07.088810 6354 consensus_queue.cc:260] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.089155 6360 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.089319 6360 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:07.089476 6361 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.089636 6361 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:07.089768 6353 tablet_bootstrap.cc:492] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: No bootstrap required, opened a new log
I20260805 23:05:07.089872 6353 ts_tablet_manager.cc:1397] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:07.089880 6354 ts_tablet_manager.cc:1428] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:07.090997 6353 raft_consensus.cc:348] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.091130 6353 raft_consensus.cc:374] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:07.091223 6353 raft_consensus.cc:729] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dc6c17624d4369ad0fad1373a1bea3, State: Initialized, Role: FOLLOWER
I20260805 23:05:07.091359 6353 consensus_queue.cc:260] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.091645 6359 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.091746 6353 ts_tablet_manager.cc:1428] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:07.091778 6359 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:07.113734 6361 raft_consensus.cc:482] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:07.113901 6361 raft_consensus.cc:504] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.115027 6361 leader_election.cc:288] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319), 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535)
I20260805 23:05:07.119366 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.115447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:49077 (local address 127.3.12.67:43535)
0805 23:05:07.115577 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.115582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.115754 (+ 172us) server_negotiation.cc:407] Connection header received
0805 23:05:07.115917 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.115920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.115970 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.116059 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:07.116711 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.117313 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.118329 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.118717 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.118831 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.118898 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.119152 (+ 254us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.119236 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260805 23:05:07.119385 6384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.115320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43535 (local address 127.3.12.66:49077)
0805 23:05:07.115661 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:07.115674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:07.115801 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:07.116171 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:07.116174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:07.116186 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:07.116300 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.116306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.117431 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.117433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.118175 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.118185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.118859 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.118863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.119025 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.119044 (+ 19us) client_negotiation.cc:769] Sending connection context
0805 23:05:07.119106 (+ 62us) client_negotiation.cc:240] Negotiation successful
0805 23:05:07.119193 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":18,"spinlock_wait_cycles":25344,"thread_start_us":95,"threads_started":1}
I20260805 23:05:07.119415 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.115444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:60671 (local address 127.3.12.65:45319)
0805 23:05:07.115577 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.115582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.115755 (+ 173us) server_negotiation.cc:407] Connection header received
0805 23:05:07.115917 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.115920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.115970 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.116059 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:07.116400 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.117094 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.118295 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.118721 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.118853 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.118914 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.119224 (+ 310us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.119302 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:05:07.119393 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.115320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:45319 (local address 127.3.12.66:60671)
0805 23:05:07.115645 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:07.115661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:07.115801 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:07.116092 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:07.116095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:07.116117 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:07.116284 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.116291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.117214 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.117218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.118145 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.118156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.118868 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.118871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.119060 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.119080 (+ 20us) client_negotiation.cc:769] Sending connection context
0805 23:05:07.119182 (+ 102us) client_negotiation.cc:240] Negotiation successful
0805 23:05:07.119249 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":79,"spinlock_wait_cycles":11776,"thread_start_us":82,"threads_started":1}
I20260805 23:05:07.119988 6255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615015a3dddd4fc68c1fffc5025854a9" candidate_uuid: "8aef2324cfdb4b2288b8798993eb5030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbc802857f34ca794d76681f52dd582" is_pre_election: true
I20260805 23:05:07.120162 6255 raft_consensus.cc:2454] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aef2324cfdb4b2288b8798993eb5030 in term 0.
I20260805 23:05:07.120203 5816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615015a3dddd4fc68c1fffc5025854a9" candidate_uuid: "8aef2324cfdb4b2288b8798993eb5030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" is_pre_election: true
I20260805 23:05:07.120492 5816 raft_consensus.cc:2454] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aef2324cfdb4b2288b8798993eb5030 in term 0.
I20260805 23:05:07.120580 5938 leader_election.cc:302] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbc802857f34ca794d76681f52dd582, 8aef2324cfdb4b2288b8798993eb5030; no voters:
I20260805 23:05:07.120906 6361 raft_consensus.cc:2781] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:07.120978 6361 raft_consensus.cc:482] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:07.121019 6361 raft_consensus.cc:3037] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:07.122428 6361 raft_consensus.cc:504] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.122829 6361 leader_election.cc:288] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [CANDIDATE]: Term 1 election: Requested vote from peers b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319), 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535)
I20260805 23:05:07.123142 5816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615015a3dddd4fc68c1fffc5025854a9" candidate_uuid: "8aef2324cfdb4b2288b8798993eb5030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1dc6c17624d4369ad0fad1373a1bea3"
I20260805 23:05:07.123261 5816 raft_consensus.cc:3037] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:07.123415 6255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615015a3dddd4fc68c1fffc5025854a9" candidate_uuid: "8aef2324cfdb4b2288b8798993eb5030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbc802857f34ca794d76681f52dd582"
I20260805 23:05:07.123579 6255 raft_consensus.cc:3037] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:07.124629 5816 raft_consensus.cc:2454] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aef2324cfdb4b2288b8798993eb5030 in term 1.
I20260805 23:05:07.124888 6255 raft_consensus.cc:2454] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aef2324cfdb4b2288b8798993eb5030 in term 1.
I20260805 23:05:07.124961 5939 leader_election.cc:302] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8aef2324cfdb4b2288b8798993eb5030, b1dc6c17624d4369ad0fad1373a1bea3; no voters:
I20260805 23:05:07.125277 6361 raft_consensus.cc:2781] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:07.125514 6361 raft_consensus.cc:686] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [term 1 LEADER]: Becoming Leader. State: Replica: 8aef2324cfdb4b2288b8798993eb5030, State: Running, Role: LEADER
I20260805 23:05:07.125923 6361 consensus_queue.cc:237] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.128319 6385 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: TxnStatusTablet state changed. Reason: New leader 8aef2324cfdb4b2288b8798993eb5030. Latest consensus state: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.128479 6385 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:07.128804 6389 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260805 23:05:07.129029 6389 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260805 23:05:07.129419 5574 catalog_manager.cc:5495] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66). New cstate: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:07.145957 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.141805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:35041 (local address 127.0.0.1:40670)
0805 23:05:07.141965 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:07.141981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:07.142083 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:07.142437 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:07.142440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:07.142448 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:07.142539 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.142543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.143601 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.143604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.144208 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.144217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.144850 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.144852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.144922 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.144969 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.145702 (+ 733us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.145714 (+ 12us) client_negotiation.cc:769] Sending connection context
0805 23:05:07.145765 (+ 51us) client_negotiation.cc:240] Negotiation successful
0805 23:05:07.145840 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260805 23:05:07.146062 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.141914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.3.12.66:35041)
0805 23:05:07.142050 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.142055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.142072 (+ 17us) server_negotiation.cc:407] Connection header received
0805 23:05:07.142231 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.142235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.142306 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.142424 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:07.142663 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.143470 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.144384 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.144732 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.144829 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.145104 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.145609 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.145722 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.145778 (+ 56us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.145837 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 23:05:07.151603 6255 raft_consensus.cc:1261] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Refusing update from remote peer 8aef2324cfdb4b2288b8798993eb5030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:07.151600 5816 raft_consensus.cc:1261] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Refusing update from remote peer 8aef2324cfdb4b2288b8798993eb5030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:07.152347 6361 consensus_queue.cc:1035] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:07.152557 6385 consensus_queue.cc:1035] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:07.156713 6360 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: TxnStatusTablet state changed. Reason: New leader 8aef2324cfdb4b2288b8798993eb5030. Latest consensus state: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.156842 6360 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:07.157716 6359 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: TxnStatusTablet state changed. Reason: New leader 8aef2324cfdb4b2288b8798993eb5030. Latest consensus state: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.157862 6359 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:07.158531 6361 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.158650 6361 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:07.158670 6396 mvcc.cc:205] Tried to move back new op lower bound from 7315337654885392384 to 7315337654790868992. Current Snapshot: MvccSnapshot[applied={T|T < 7315337654885392384}]
I20260805 23:05:07.159550 6359 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.159652 6359 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P b1dc6c17624d4369ad0fad1373a1bea3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:07.161396 6398 mvcc.cc:205] Tried to move back new op lower bound from 7315337654885392384 to 7315337654790868992. Current Snapshot: MvccSnapshot[applied={T|T < 7315337654885392384}]
I20260805 23:05:07.166363 6385 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.166493 6385 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P 8aef2324cfdb4b2288b8798993eb5030: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:07.168046 6360 tablet_replica.cc:439] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } } }
I20260805 23:05:07.168207 6360 tablet_replica.cc:442] T 615015a3dddd4fc68c1fffc5025854a9 P 0bbc802857f34ca794d76681f52dd582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:07.168318 6400 mvcc.cc:205] Tried to move back new op lower bound from 7315337654885392384 to 7315337654790868992. Current Snapshot: MvccSnapshot[applied={T|T < 7315337654885392384}]
I20260805 23:05:07.173100 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.168550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.3.12.65:45319)
0805 23:05:07.168872 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.168877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.168891 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:07.168972 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.168975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.169038 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.169156 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:07.169460 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.170266 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.171535 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.171761 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.171848 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.172202 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.172671 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.172760 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.172882 (+ 122us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.172936 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260805 23:05:07.184903 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.180933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:35041 (local address 127.3.12.65:43279)
0805 23:05:07.181357 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:07.181369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:07.181463 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:07.181699 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:07.181701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:07.181707 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:07.181823 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.181828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.182825 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.182827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.183547 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.183557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.184132 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.184135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.184210 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.184309 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.184633 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.184644 (+ 11us) client_negotiation.cc:769] Sending connection context
0805 23:05:07.184696 (+ 52us) client_negotiation.cc:240] Negotiation successful
0805 23:05:07.184762 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":118,"threads_started":1}
I20260805 23:05:07.184897 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.181039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:43279 (local address 127.3.12.66:35041)
0805 23:05:07.181183 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.181186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.181457 (+ 271us) server_negotiation.cc:407] Connection header received
0805 23:05:07.181541 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.181545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.181601 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.181683 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:07.181944 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.182720 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.183667 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.184019 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.184126 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.184460 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.184546 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.184622 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.184707 (+ 85us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.184760 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260805 23:05:07.193034 6255 raft_consensus.cc:1261] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Refusing update from remote peer b1dc6c17624d4369ad0fad1373a1bea3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:07.193223 6034 raft_consensus.cc:1261] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Refusing update from remote peer b1dc6c17624d4369ad0fad1373a1bea3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:07.193553 6359 consensus_queue.cc:1035] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:07.193755 6368 consensus_queue.cc:1035] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:07.218223 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.214108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:56167 (local address 127.3.12.65:45319)
0805 23:05:07.214238 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.214244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.214264 (+ 20us) server_negotiation.cc:407] Connection header received
0805 23:05:07.214412 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.214414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.214457 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.214521 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:07.214841 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.215539 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.217104 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.217328 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.217412 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.217751 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.217818 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.217869 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.218016 (+ 147us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.218076 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:05:07.218246 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.213827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:45319 (local address 127.3.12.66:56167)
0805 23:05:07.214192 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:07.214209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:07.214309 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:07.214557 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:07.214560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:07.214568 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:07.214722 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.214729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.215882 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.215885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.216974 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.216988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.217446 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.217450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.217551 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.217613 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.217903 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.217910 (+ 7us) client_negotiation.cc:769] Sending connection context
0805 23:05:07.217986 (+ 76us) client_negotiation.cc:240] Negotiation successful
0805 23:05:07.218073 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":117,"threads_started":1}
I20260805 23:05:07.254494 5574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:07.254736 5574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:07.257773 5574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: Sending DeleteTablet for 3 replicas of tablet 9e989df8c9364bcd997478d83f138dec
I20260805 23:05:07.258807 6217 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e989df8c9364bcd997478d83f138dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:07 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:54384
I20260805 23:05:07.259075 5788 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e989df8c9364bcd997478d83f138dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:07 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:57264
I20260805 23:05:07.259645 5996 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e989df8c9364bcd997478d83f138dec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:07 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:40660
I20260805 23:05:07.260324 6424 tablet_replica.cc:330] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: stopping tablet replica
I20260805 23:05:07.260392 5574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:07.260842 5574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:07.260982 6424 raft_consensus.cc:2229] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:07.261271 6424 raft_consensus.cc:2258] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:07.261716 6428 tablet_replica.cc:330] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: stopping tablet replica
I20260805 23:05:07.263582 6424 ts_tablet_manager.cc:1905] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:07.266255 6217 tablet_service.cc:1467] Processing CreateTablet for tablet c934ccf23d7a4723a82e7d5545fe59e0 (DEFAULT_TABLE table=test-table [id=9a2551c045324f71a0f78e22307f597d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:07.266562 6217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c934ccf23d7a4723a82e7d5545fe59e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:07.266827 6424 ts_tablet_manager.cc:1918] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:07.266921 6424 log.cc:1196] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/9e989df8c9364bcd997478d83f138dec
I20260805 23:05:07.267325 6424 ts_tablet_manager.cc:1939] T 9e989df8c9364bcd997478d83f138dec P 0bbc802857f34ca794d76681f52dd582: Deleting consensus metadata
I20260805 23:05:07.268860 5555 catalog_manager.cc:4841] TS 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535): tablet 9e989df8c9364bcd997478d83f138dec (table test-table [id=3d5b3f1d800f418a84a2320f31d0044a]) successfully deleted
I20260805 23:05:07.269068 5996 tablet_service.cc:1467] Processing CreateTablet for tablet c934ccf23d7a4723a82e7d5545fe59e0 (DEFAULT_TABLE table=test-table [id=9a2551c045324f71a0f78e22307f597d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:07.269361 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c934ccf23d7a4723a82e7d5545fe59e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:07.271836 6354 tablet_bootstrap.cc:492] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: Bootstrap starting.
I20260805 23:05:07.272931 6354 tablet_bootstrap.cc:654] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:07.273166 6355 tablet_bootstrap.cc:492] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: Bootstrap starting.
I20260805 23:05:07.274140 6355 tablet_bootstrap.cc:654] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:07.275187 6354 tablet_bootstrap.cc:492] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: No bootstrap required, opened a new log
I20260805 23:05:07.275281 6354 ts_tablet_manager.cc:1397] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260805 23:05:07.275452 5788 tablet_service.cc:1467] Processing CreateTablet for tablet c934ccf23d7a4723a82e7d5545fe59e0 (DEFAULT_TABLE table=test-table [id=9a2551c045324f71a0f78e22307f597d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:07.275740 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c934ccf23d7a4723a82e7d5545fe59e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:07.275735 6354 raft_consensus.cc:348] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.275848 6354 raft_consensus.cc:374] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:07.275880 6354 raft_consensus.cc:729] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aef2324cfdb4b2288b8798993eb5030, State: Initialized, Role: FOLLOWER
I20260805 23:05:07.276000 6354 consensus_queue.cc:260] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.276224 6354 ts_tablet_manager.cc:1428] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260805 23:05:07.277323 6427 tablet_replica.cc:330] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: stopping tablet replica
I20260805 23:05:07.277743 6427 raft_consensus.cc:2229] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:07.278151 6427 raft_consensus.cc:2258] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:07.278571 6353 tablet_bootstrap.cc:492] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: Bootstrap starting.
I20260805 23:05:07.278861 6428 raft_consensus.cc:2229] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:07.279023 6428 raft_consensus.cc:2258] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:07.279601 6353 tablet_bootstrap.cc:654] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:07.280488 6428 ts_tablet_manager.cc:1905] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:07.282614 6353 tablet_bootstrap.cc:492] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: No bootstrap required, opened a new log
I20260805 23:05:07.282719 6353 ts_tablet_manager.cc:1397] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260805 23:05:07.283265 6353 raft_consensus.cc:348] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.283385 6353 raft_consensus.cc:374] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:07.283420 6353 raft_consensus.cc:729] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dc6c17624d4369ad0fad1373a1bea3, State: Initialized, Role: FOLLOWER
I20260805 23:05:07.283579 6428 ts_tablet_manager.cc:1918] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:07.283679 6428 log.cc:1196] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/9e989df8c9364bcd997478d83f138dec
I20260805 23:05:07.284003 6428 ts_tablet_manager.cc:1939] T 9e989df8c9364bcd997478d83f138dec P 8aef2324cfdb4b2288b8798993eb5030: Deleting consensus metadata
I20260805 23:05:07.284937 6353 consensus_queue.cc:260] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.285332 6353 ts_tablet_manager.cc:1428] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260805 23:05:07.285372 5554 catalog_manager.cc:4841] TS 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041): tablet 9e989df8c9364bcd997478d83f138dec (table test-table [id=3d5b3f1d800f418a84a2320f31d0044a]) successfully deleted
I20260805 23:05:07.286623 6355 tablet_bootstrap.cc:492] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: No bootstrap required, opened a new log
I20260805 23:05:07.286722 6355 ts_tablet_manager.cc:1397] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260805 23:05:07.286731 6427 ts_tablet_manager.cc:1905] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:07.287215 6355 raft_consensus.cc:348] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.287338 6355 raft_consensus.cc:374] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:07.287372 6355 raft_consensus.cc:729] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbc802857f34ca794d76681f52dd582, State: Initialized, Role: FOLLOWER
I20260805 23:05:07.289953 6427 ts_tablet_manager.cc:1918] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:07.290057 6427 log.cc:1196] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/9e989df8c9364bcd997478d83f138dec
I20260805 23:05:07.290380 6427 ts_tablet_manager.cc:1939] T 9e989df8c9364bcd997478d83f138dec P b1dc6c17624d4369ad0fad1373a1bea3: Deleting consensus metadata
I20260805 23:05:07.291534 5557 catalog_manager.cc:4841] TS b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319): tablet 9e989df8c9364bcd997478d83f138dec (table test-table [id=3d5b3f1d800f418a84a2320f31d0044a]) successfully deleted
I20260805 23:05:07.292032 6355 consensus_queue.cc:260] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.293128 6355 ts_tablet_manager.cc:1428] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260805 23:05:07.656802 6385 raft_consensus.cc:482] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:07.656958 6385 raft_consensus.cc:504] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.657418 6385 leader_election.cc:288] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319), 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535)
I20260805 23:05:07.657904 5816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c934ccf23d7a4723a82e7d5545fe59e0" candidate_uuid: "8aef2324cfdb4b2288b8798993eb5030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" is_pre_election: true
I20260805 23:05:07.657953 6249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c934ccf23d7a4723a82e7d5545fe59e0" candidate_uuid: "8aef2324cfdb4b2288b8798993eb5030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbc802857f34ca794d76681f52dd582" is_pre_election: true
I20260805 23:05:07.658092 5816 raft_consensus.cc:2454] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aef2324cfdb4b2288b8798993eb5030 in term 0.
I20260805 23:05:07.658092 6249 raft_consensus.cc:2454] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aef2324cfdb4b2288b8798993eb5030 in term 0.
I20260805 23:05:07.658416 5939 leader_election.cc:302] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8aef2324cfdb4b2288b8798993eb5030, b1dc6c17624d4369ad0fad1373a1bea3; no voters:
I20260805 23:05:07.658640 6385 raft_consensus.cc:2781] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:07.658703 6385 raft_consensus.cc:482] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:07.658730 6385 raft_consensus.cc:3037] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:07.659822 6385 raft_consensus.cc:504] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.660192 6385 leader_election.cc:288] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [CANDIDATE]: Term 1 election: Requested vote from peers b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319), 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535)
I20260805 23:05:07.660607 6249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c934ccf23d7a4723a82e7d5545fe59e0" candidate_uuid: "8aef2324cfdb4b2288b8798993eb5030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbc802857f34ca794d76681f52dd582"
I20260805 23:05:07.660607 5816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c934ccf23d7a4723a82e7d5545fe59e0" candidate_uuid: "8aef2324cfdb4b2288b8798993eb5030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1dc6c17624d4369ad0fad1373a1bea3"
I20260805 23:05:07.660748 5816 raft_consensus.cc:3037] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:07.660748 6249 raft_consensus.cc:3037] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:07.662034 6249 raft_consensus.cc:2454] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aef2324cfdb4b2288b8798993eb5030 in term 1.
I20260805 23:05:07.662034 5816 raft_consensus.cc:2454] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aef2324cfdb4b2288b8798993eb5030 in term 1.
I20260805 23:05:07.662360 5938 leader_election.cc:302] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbc802857f34ca794d76681f52dd582, 8aef2324cfdb4b2288b8798993eb5030; no voters:
I20260805 23:05:07.662559 6385 raft_consensus.cc:2781] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:07.662664 6385 raft_consensus.cc:686] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 1 LEADER]: Becoming Leader. State: Replica: 8aef2324cfdb4b2288b8798993eb5030, State: Running, Role: LEADER
I20260805 23:05:07.662842 6385 consensus_queue.cc:237] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:07.664687 5574 catalog_manager.cc:5495] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66). New cstate: current_term: 1 leader_uuid: "8aef2324cfdb4b2288b8798993eb5030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:07.754325 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.750658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.3.12.66:35041)
0805 23:05:07.750899 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.750906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.750961 (+ 55us) server_negotiation.cc:407] Connection header received
0805 23:05:07.751134 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.751137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.751189 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.751284 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:07.751582 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.752201 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.753129 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.753383 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.753510 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.753782 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.753867 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.753948 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.754070 (+ 122us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.754128 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260805 23:05:07.763693 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.760722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:56297 (local address 127.3.12.66:35041)
0805 23:05:07.760830 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:07.760833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:07.761055 (+ 222us) server_negotiation.cc:407] Connection header received
0805 23:05:07.761184 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:07.761186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:07.761226 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:07.761290 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:07.761501 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.761979 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.762806 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.763016 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.763097 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.763306 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.763371 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.763424 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:07.763504 (+ 80us) server_negotiation.cc:299] Negotiation successful
0805 23:05:07.763548 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260805 23:05:07.763708 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:07.760661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:35041 (local address 127.3.12.66:56297)
0805 23:05:07.760968 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:07.760981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:07.761086 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:07.761316 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:07.761318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:07.761324 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:07.761413 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.761417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.762090 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.762092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.762701 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:07.762709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:07.763119 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:07.763122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:07.763182 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:07.763206 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:07.763444 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:07.763449 (+ 5us) client_negotiation.cc:769] Sending connection context
0805 23:05:07.763497 (+ 48us) client_negotiation.cc:240] Negotiation successful
0805 23:05:07.763577 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":138,"threads_started":1}
I20260805 23:05:07.770044 6249 raft_consensus.cc:1261] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Refusing update from remote peer 8aef2324cfdb4b2288b8798993eb5030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:07.770059 5816 raft_consensus.cc:1261] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Refusing update from remote peer 8aef2324cfdb4b2288b8798993eb5030: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:07.770529 6361 consensus_queue.cc:1035] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:07.770721 6385 consensus_queue.cc:1035] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 23:05:07.786100 6434 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260805 23:05:09.346087 5574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:09.346241 5574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:09.348934 5574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: Sending DeleteTablet for 3 replicas of tablet c934ccf23d7a4723a82e7d5545fe59e0
I20260805 23:05:09.349696 5788 tablet_service.cc:1514] Processing DeleteTablet for tablet c934ccf23d7a4723a82e7d5545fe59e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:09 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:57264
I20260805 23:05:09.349759 6217 tablet_service.cc:1514] Processing DeleteTablet for tablet c934ccf23d7a4723a82e7d5545fe59e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:09 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:54384
I20260805 23:05:09.350072 5996 tablet_service.cc:1514] Processing DeleteTablet for tablet c934ccf23d7a4723a82e7d5545fe59e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:09 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:40660
I20260805 23:05:09.350117 6490 tablet_replica.cc:330] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: stopping tablet replica
I20260805 23:05:09.350255 6490 raft_consensus.cc:2229] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:09.350387 6490 raft_consensus.cc:2258] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:09.350468 6495 tablet_replica.cc:330] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: stopping tablet replica
I20260805 23:05:09.350669 6495 raft_consensus.cc:2229] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:09.351063 6495 raft_consensus.cc:2258] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:09.351948 6490 ts_tablet_manager.cc:1905] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:09.354382 6495 ts_tablet_manager.cc:1905] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:09.355222 6490 ts_tablet_manager.cc:1918] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 23:05:09.355307 6490 log.cc:1196] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/c934ccf23d7a4723a82e7d5545fe59e0
I20260805 23:05:09.355645 6490 ts_tablet_manager.cc:1939] T c934ccf23d7a4723a82e7d5545fe59e0 P b1dc6c17624d4369ad0fad1373a1bea3: Deleting consensus metadata
I20260805 23:05:09.356420 5557 catalog_manager.cc:4841] TS b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319): tablet c934ccf23d7a4723a82e7d5545fe59e0 (table test-table [id=9a2551c045324f71a0f78e22307f597d]) successfully deleted
I20260805 23:05:09.357596 6495 ts_tablet_manager.cc:1918] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 23:05:09.357699 6495 log.cc:1196] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/c934ccf23d7a4723a82e7d5545fe59e0
I20260805 23:05:09.357977 6495 ts_tablet_manager.cc:1939] T c934ccf23d7a4723a82e7d5545fe59e0 P 8aef2324cfdb4b2288b8798993eb5030: Deleting consensus metadata
I20260805 23:05:09.358570 6492 tablet_replica.cc:330] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: stopping tablet replica
I20260805 23:05:09.358690 5554 catalog_manager.cc:4841] TS 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041): tablet c934ccf23d7a4723a82e7d5545fe59e0 (table test-table [id=9a2551c045324f71a0f78e22307f597d]) successfully deleted
I20260805 23:05:09.358776 6492 raft_consensus.cc:2229] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:09.358940 6492 raft_consensus.cc:2258] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:09.359588 6492 ts_tablet_manager.cc:1905] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:09.361899 6492 ts_tablet_manager.cc:1918] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 23:05:09.361999 6492 log.cc:1196] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/c934ccf23d7a4723a82e7d5545fe59e0
I20260805 23:05:09.362284 6492 ts_tablet_manager.cc:1939] T c934ccf23d7a4723a82e7d5545fe59e0 P 0bbc802857f34ca794d76681f52dd582: Deleting consensus metadata
I20260805 23:05:09.363133 5555 catalog_manager.cc:4841] TS 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535): tablet c934ccf23d7a4723a82e7d5545fe59e0 (table test-table [id=9a2551c045324f71a0f78e22307f597d]) successfully deleted
Aug 05 23:05:09 dist-test-slave-2d19 krb5kdc[5505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 23:05:09 dist-test-slave-2d19 krb5kdc[5505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:09.385382 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.375726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.3.12.126:34095)
0805 23:05:09.376030 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:09.376041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:09.376185 (+ 144us) server_negotiation.cc:407] Connection header received
0805 23:05:09.376434 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:09.376437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:09.376508 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:09.376615 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:09.377484 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.378032 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.378910 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.379150 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.379261 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.382058 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:09.382061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:09.382063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:09.382089 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:09.383596 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:09.384088 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:09.384091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:09.384092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:09.384189 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:09.384555 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:09.384557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:09.384558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:09.384747 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:09.384858 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:09.385116 (+ 258us) server_negotiation.cc:299] Negotiation successful
0805 23:05:09.385243 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
I20260805 23:05:09.388144 5574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:09.388763 5574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:09.393807 5788 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb33312afc444cd97ac803909ce7bda (DEFAULT_TABLE table=test-table [id=bb05635413904abda0eeeb63223745ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:09.393880 5996 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb33312afc444cd97ac803909ce7bda (DEFAULT_TABLE table=test-table [id=bb05635413904abda0eeeb63223745ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:09.394171 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb33312afc444cd97ac803909ce7bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:09.394172 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb33312afc444cd97ac803909ce7bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:09.394150 6217 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb33312afc444cd97ac803909ce7bda (DEFAULT_TABLE table=test-table [id=bb05635413904abda0eeeb63223745ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:09.394392 6217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb33312afc444cd97ac803909ce7bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:09.397143 6515 tablet_bootstrap.cc:492] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: Bootstrap starting.
I20260805 23:05:09.397178 6516 tablet_bootstrap.cc:492] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: Bootstrap starting.
I20260805 23:05:09.397517 6517 tablet_bootstrap.cc:492] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: Bootstrap starting.
I20260805 23:05:09.398082 6515 tablet_bootstrap.cc:654] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:09.398290 6516 tablet_bootstrap.cc:654] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:09.398447 6517 tablet_bootstrap.cc:654] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:09.399626 6515 tablet_bootstrap.cc:492] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: No bootstrap required, opened a new log
I20260805 23:05:09.399735 6515 ts_tablet_manager.cc:1397] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:09.400084 6517 tablet_bootstrap.cc:492] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: No bootstrap required, opened a new log
I20260805 23:05:09.400161 6517 ts_tablet_manager.cc:1397] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:09.400174 6515 raft_consensus.cc:348] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.400331 6515 raft_consensus.cc:374] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:09.400381 6515 raft_consensus.cc:729] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbc802857f34ca794d76681f52dd582, State: Initialized, Role: FOLLOWER
I20260805 23:05:09.400527 6515 consensus_queue.cc:260] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.400658 6517 raft_consensus.cc:348] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.400786 6517 raft_consensus.cc:374] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:09.400795 6516 tablet_bootstrap.cc:492] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: No bootstrap required, opened a new log
I20260805 23:05:09.400833 6517 raft_consensus.cc:729] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dc6c17624d4369ad0fad1373a1bea3, State: Initialized, Role: FOLLOWER
I20260805 23:05:09.400879 6516 ts_tablet_manager.cc:1397] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260805 23:05:09.400941 6515 ts_tablet_manager.cc:1428] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260805 23:05:09.401281 6516 raft_consensus.cc:348] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.401405 6516 raft_consensus.cc:374] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:09.401438 6516 raft_consensus.cc:729] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aef2324cfdb4b2288b8798993eb5030, State: Initialized, Role: FOLLOWER
I20260805 23:05:09.401382 6517 consensus_queue.cc:260] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.401579 6516 consensus_queue.cc:260] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.401880 6516 ts_tablet_manager.cc:1428] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:09.401839 6517 ts_tablet_manager.cc:1428] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:09.430943 6523 raft_consensus.cc:482] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:09.431099 6523 raft_consensus.cc:504] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.432363 6523 leader_election.cc:288] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319), 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041)
I20260805 23:05:09.438488 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.432640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:55275 (local address 127.3.12.65:45319)
0805 23:05:09.433045 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:09.433059 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:09.433322 (+ 263us) server_negotiation.cc:407] Connection header received
0805 23:05:09.433596 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:09.433600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:09.433656 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:09.433762 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:09.434220 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.435104 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.437153 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.437778 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.437908 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.437998 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:09.438241 (+ 243us) server_negotiation.cc:299] Negotiation successful
0805 23:05:09.438331 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":178,"threads_started":1}
I20260805 23:05:09.438392 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.432546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:45319 (local address 127.3.12.67:55275)
0805 23:05:09.433196 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:09.433216 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:09.433426 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:09.433834 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:09.433838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:09.433884 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:09.434078 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:09.434087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.435251 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.435253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:09.436989 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:09.437004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.437927 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.437932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:09.438084 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.438112 (+ 28us) client_negotiation.cc:769] Sending connection context
0805 23:05:09.438192 (+ 80us) client_negotiation.cc:240] Negotiation successful
0805 23:05:09.438251 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":187,"spinlock_wait_cycles":10112,"thread_start_us":285,"threads_started":1}
I20260805 23:05:09.438719 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.432615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:35041 (local address 127.3.12.67:53511)
0805 23:05:09.433432 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:09.433444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:09.433546 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:09.433834 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:09.433838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:09.433883 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:09.434098 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:09.434103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.435234 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.435239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:09.437006 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:09.437014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.438149 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.438152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:09.438281 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.438304 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:05:09.438377 (+ 73us) client_negotiation.cc:240] Negotiation successful
0805 23:05:09.438447 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":242,"spinlock_wait_cycles":30336,"thread_start_us":480,"threads_started":1}
I20260805 23:05:09.439100 5814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb33312afc444cd97ac803909ce7bda" candidate_uuid: "0bbc802857f34ca794d76681f52dd582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" is_pre_election: true
I20260805 23:05:09.439172 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.432908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:53511 (local address 127.3.12.66:35041)
0805 23:05:09.433332 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:09.433340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:09.433538 (+ 198us) server_negotiation.cc:407] Connection header received
0805 23:05:09.433628 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:09.433631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:09.433706 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:09.433794 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:09.434220 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.435104 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.437143 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.437889 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.437981 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.438074 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:09.438514 (+ 440us) server_negotiation.cc:299] Negotiation successful
0805 23:05:09.439014 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":182,"threads_started":1}
I20260805 23:05:09.439280 5814 raft_consensus.cc:2454] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bbc802857f34ca794d76681f52dd582 in term 0.
I20260805 23:05:09.439584 6034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb33312afc444cd97ac803909ce7bda" candidate_uuid: "0bbc802857f34ca794d76681f52dd582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aef2324cfdb4b2288b8798993eb5030" is_pre_election: true
I20260805 23:05:09.439733 6034 raft_consensus.cc:2454] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bbc802857f34ca794d76681f52dd582 in term 0.
I20260805 23:05:09.439761 6150 leader_election.cc:302] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbc802857f34ca794d76681f52dd582, b1dc6c17624d4369ad0fad1373a1bea3; no voters:
I20260805 23:05:09.440096 6523 raft_consensus.cc:2781] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:09.440172 6523 raft_consensus.cc:482] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:09.440229 6523 raft_consensus.cc:3037] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:09.441323 6523 raft_consensus.cc:504] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.441697 6523 leader_election.cc:288] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [CANDIDATE]: Term 1 election: Requested vote from peers b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319), 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041)
I20260805 23:05:09.442095 6034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb33312afc444cd97ac803909ce7bda" candidate_uuid: "0bbc802857f34ca794d76681f52dd582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aef2324cfdb4b2288b8798993eb5030"
I20260805 23:05:09.442094 5814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb33312afc444cd97ac803909ce7bda" candidate_uuid: "0bbc802857f34ca794d76681f52dd582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1dc6c17624d4369ad0fad1373a1bea3"
I20260805 23:05:09.442232 6034 raft_consensus.cc:3037] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:09.442233 5814 raft_consensus.cc:3037] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:09.443413 5814 raft_consensus.cc:2454] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bbc802857f34ca794d76681f52dd582 in term 1.
I20260805 23:05:09.443413 6034 raft_consensus.cc:2454] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bbc802857f34ca794d76681f52dd582 in term 1.
I20260805 23:05:09.443763 6148 leader_election.cc:302] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbc802857f34ca794d76681f52dd582, 8aef2324cfdb4b2288b8798993eb5030; no voters:
I20260805 23:05:09.443939 6523 raft_consensus.cc:2781] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:09.444178 6523 raft_consensus.cc:686] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 1 LEADER]: Becoming Leader. State: Replica: 0bbc802857f34ca794d76681f52dd582, State: Running, Role: LEADER
I20260805 23:05:09.444481 6523 consensus_queue.cc:237] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.446945 5575 catalog_manager.cc:5495] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bbc802857f34ca794d76681f52dd582 (127.3.12.67). New cstate: current_term: 1 leader_uuid: "0bbc802857f34ca794d76681f52dd582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } health_report { overall_health: HEALTHY } } }
I20260805 23:05:09.474519 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.470377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.3.12.67:43535)
0805 23:05:09.470682 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:09.470690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:09.470704 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:09.470784 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:09.470786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:09.470842 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:09.470960 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:09.471250 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.471853 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.472810 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.473110 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.473204 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.473545 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:09.474158 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:09.474260 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:09.474327 (+ 67us) server_negotiation.cc:299] Negotiation successful
0805 23:05:09.474383 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":151,"threads_started":1}
I20260805 23:05:09.485024 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.481799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:44707 (local address 127.3.12.66:35041)
0805 23:05:09.481958 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:09.481962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:09.481977 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:09.482109 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:09.482111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:09.482166 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:09.482235 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:09.482597 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.483087 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.483942 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.484160 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.484237 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.484548 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:09.484621 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:09.484683 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:09.484855 (+ 172us) server_negotiation.cc:299] Negotiation successful
0805 23:05:09.484906 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 23:05:09.485071 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.481533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:35041 (local address 127.3.12.67:44707)
0805 23:05:09.481909 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:09.481924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:09.482018 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:09.482350 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:09.482353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:09.482363 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:09.482490 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:09.482496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.483190 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.483193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:09.483838 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:09.483847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.484296 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.484299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:09.484368 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.484413 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:09.484740 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:09.484750 (+ 10us) client_negotiation.cc:769] Sending connection context
0805 23:05:09.484834 (+ 84us) client_negotiation.cc:240] Negotiation successful
0805 23:05:09.484910 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":133,"threads_started":1}
I20260805 23:05:09.492673 6034 raft_consensus.cc:1261] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Refusing update from remote peer 0bbc802857f34ca794d76681f52dd582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:09.492707 5814 raft_consensus.cc:1261] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Refusing update from remote peer 0bbc802857f34ca794d76681f52dd582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:09.493275 6530 consensus_queue.cc:1035] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:09.493476 6523 consensus_queue.cc:1035] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:09.513088 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.509533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:35895 (local address 127.3.12.67:43535)
0805 23:05:09.509845 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:09.509850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:09.509929 (+ 79us) server_negotiation.cc:407] Connection header received
0805 23:05:09.510095 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:09.510097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:09.510142 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:09.510212 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:09.510380 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.510865 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.511795 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.512137 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.512299 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.512580 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:09.512680 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:09.512786 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:09.512879 (+ 93us) server_negotiation.cc:299] Negotiation successful
0805 23:05:09.512939 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260805 23:05:09.513092 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.509385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43535 (local address 127.3.12.66:35895)
0805 23:05:09.509827 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:09.509842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:09.509968 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:09.510218 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:09.510220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:09.510225 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:09.510316 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:09.510320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.510947 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.510949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:09.511662 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:09.511672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.512301 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.512304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:09.512395 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.512444 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:09.512791 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:09.512801 (+ 10us) client_negotiation.cc:769] Sending connection context
0805 23:05:09.512869 (+ 68us) client_negotiation.cc:240] Negotiation successful
0805 23:05:09.512960 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":195,"threads_started":1}
I20260805 23:05:09.536892 5577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:09.537088 5577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:09.539497 5577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: Sending DeleteTablet for 3 replicas of tablet 9eb33312afc444cd97ac803909ce7bda
I20260805 23:05:09.540150 5788 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb33312afc444cd97ac803909ce7bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:09 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:57264
I20260805 23:05:09.540313 5996 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb33312afc444cd97ac803909ce7bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:09 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:40660
I20260805 23:05:09.540437 6490 tablet_replica.cc:330] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: stopping tablet replica
I20260805 23:05:09.540436 6217 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb33312afc444cd97ac803909ce7bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:09 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:54384
I20260805 23:05:09.540594 6490 raft_consensus.cc:2229] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:09.540652 6492 tablet_replica.cc:330] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: stopping tablet replica
I20260805 23:05:09.540727 6490 raft_consensus.cc:2258] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:09.540859 6492 raft_consensus.cc:2229] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:09.540925 6495 tablet_replica.cc:330] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: stopping tablet replica
I20260805 23:05:09.541049 6495 raft_consensus.cc:2229] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:09.541143 6495 raft_consensus.cc:2258] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:09.541379 6492 raft_consensus.cc:2258] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:09.541949 6495 ts_tablet_manager.cc:1905] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:09.542510 6492 ts_tablet_manager.cc:1905] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:09.542456 5577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:09.542872 5577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:09.545315 6495 ts_tablet_manager.cc:1918] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:09.545418 6495 log.cc:1196] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/9eb33312afc444cd97ac803909ce7bda
I20260805 23:05:09.545665 6495 ts_tablet_manager.cc:1939] T 9eb33312afc444cd97ac803909ce7bda P 8aef2324cfdb4b2288b8798993eb5030: Deleting consensus metadata
I20260805 23:05:09.545881 6492 ts_tablet_manager.cc:1918] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:09.546016 6492 log.cc:1196] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/9eb33312afc444cd97ac803909ce7bda
I20260805 23:05:09.546381 6492 ts_tablet_manager.cc:1939] T 9eb33312afc444cd97ac803909ce7bda P 0bbc802857f34ca794d76681f52dd582: Deleting consensus metadata
I20260805 23:05:09.546473 5554 catalog_manager.cc:4841] TS 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041): tablet 9eb33312afc444cd97ac803909ce7bda (table test-table [id=bb05635413904abda0eeeb63223745ea]) successfully deleted
I20260805 23:05:09.549475 6490 ts_tablet_manager.cc:1905] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:09.549774 5996 tablet_service.cc:1467] Processing CreateTablet for tablet 7a39d11c72904d9bb90ef86603ca766c (DEFAULT_TABLE table=test-table [id=01492e1aa6d944e7b8cd040b3c4aaf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:09.550026 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a39d11c72904d9bb90ef86603ca766c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:09.552642 6516 tablet_bootstrap.cc:492] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: Bootstrap starting.
I20260805 23:05:09.552765 6490 ts_tablet_manager.cc:1918] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:09.552870 6490 log.cc:1196] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/9eb33312afc444cd97ac803909ce7bda
I20260805 23:05:09.553155 6490 ts_tablet_manager.cc:1939] T 9eb33312afc444cd97ac803909ce7bda P b1dc6c17624d4369ad0fad1373a1bea3: Deleting consensus metadata
I20260805 23:05:09.553545 6516 tablet_bootstrap.cc:654] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:09.553722 5788 tablet_service.cc:1467] Processing CreateTablet for tablet 7a39d11c72904d9bb90ef86603ca766c (DEFAULT_TABLE table=test-table [id=01492e1aa6d944e7b8cd040b3c4aaf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:09.553982 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a39d11c72904d9bb90ef86603ca766c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:09.553989 5557 catalog_manager.cc:4841] TS b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319): tablet 9eb33312afc444cd97ac803909ce7bda (table test-table [id=bb05635413904abda0eeeb63223745ea]) successfully deleted
I20260805 23:05:09.554497 6217 tablet_service.cc:1467] Processing CreateTablet for tablet 7a39d11c72904d9bb90ef86603ca766c (DEFAULT_TABLE table=test-table [id=01492e1aa6d944e7b8cd040b3c4aaf68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:09.554771 6217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a39d11c72904d9bb90ef86603ca766c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:09.556890 5555 catalog_manager.cc:4841] TS 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535): tablet 9eb33312afc444cd97ac803909ce7bda (table test-table [id=bb05635413904abda0eeeb63223745ea]) successfully deleted
I20260805 23:05:09.557263 6517 tablet_bootstrap.cc:492] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3: Bootstrap starting.
I20260805 23:05:09.557787 6515 tablet_bootstrap.cc:492] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: Bootstrap starting.
I20260805 23:05:09.558531 6517 tablet_bootstrap.cc:654] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:09.558871 6515 tablet_bootstrap.cc:654] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:09.560222 6517 tablet_bootstrap.cc:492] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3: No bootstrap required, opened a new log
I20260805 23:05:09.560339 6517 ts_tablet_manager.cc:1397] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:09.560868 6517 raft_consensus.cc:348] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.560990 6517 raft_consensus.cc:374] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:09.561023 6517 raft_consensus.cc:729] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dc6c17624d4369ad0fad1373a1bea3, State: Initialized, Role: FOLLOWER
I20260805 23:05:09.561223 6515 tablet_bootstrap.cc:492] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: No bootstrap required, opened a new log
I20260805 23:05:09.561168 6517 consensus_queue.cc:260] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.561299 6515 ts_tablet_manager.cc:1397] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260805 23:05:09.561467 6517 ts_tablet_manager.cc:1428] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260805 23:05:09.561790 6515 raft_consensus.cc:348] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.561914 6515 raft_consensus.cc:374] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:09.561955 6515 raft_consensus.cc:729] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbc802857f34ca794d76681f52dd582, State: Initialized, Role: FOLLOWER
I20260805 23:05:09.562115 6515 consensus_queue.cc:260] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.562407 6515 ts_tablet_manager.cc:1428] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:09.569136 6516 tablet_bootstrap.cc:492] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: No bootstrap required, opened a new log
I20260805 23:05:09.569258 6516 ts_tablet_manager.cc:1397] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260805 23:05:09.569741 6516 raft_consensus.cc:348] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.569868 6516 raft_consensus.cc:374] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:09.569908 6516 raft_consensus.cc:729] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aef2324cfdb4b2288b8798993eb5030, State: Initialized, Role: FOLLOWER
I20260805 23:05:09.570048 6516 consensus_queue.cc:260] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.570334 6516 ts_tablet_manager.cc:1428] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260805 23:05:09.674068 6525 raft_consensus.cc:482] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:09.674273 6525 raft_consensus.cc:504] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.674865 6525 leader_election.cc:288] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041), 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535)
I20260805 23:05:09.675403 6255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a39d11c72904d9bb90ef86603ca766c" candidate_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbc802857f34ca794d76681f52dd582" is_pre_election: true
I20260805 23:05:09.675421 6034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a39d11c72904d9bb90ef86603ca766c" candidate_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aef2324cfdb4b2288b8798993eb5030" is_pre_election: true
I20260805 23:05:09.675602 6255 raft_consensus.cc:2454] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1dc6c17624d4369ad0fad1373a1bea3 in term 0.
I20260805 23:05:09.675630 6034 raft_consensus.cc:2454] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1dc6c17624d4369ad0fad1373a1bea3 in term 0.
I20260805 23:05:09.676100 5709 leader_election.cc:302] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8aef2324cfdb4b2288b8798993eb5030, b1dc6c17624d4369ad0fad1373a1bea3; no voters:
I20260805 23:05:09.676328 6525 raft_consensus.cc:2781] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:09.676414 6525 raft_consensus.cc:482] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:09.676513 6525 raft_consensus.cc:3037] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:09.677728 6525 raft_consensus.cc:504] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.678267 6525 leader_election.cc:288] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [CANDIDATE]: Term 1 election: Requested vote from peers 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041), 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535)
I20260805 23:05:09.678637 6034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a39d11c72904d9bb90ef86603ca766c" candidate_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8aef2324cfdb4b2288b8798993eb5030"
I20260805 23:05:09.678689 6255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a39d11c72904d9bb90ef86603ca766c" candidate_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbc802857f34ca794d76681f52dd582"
I20260805 23:05:09.678792 6034 raft_consensus.cc:3037] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:09.678835 6255 raft_consensus.cc:3037] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:09.680094 6255 raft_consensus.cc:2454] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1dc6c17624d4369ad0fad1373a1bea3 in term 1.
I20260805 23:05:09.680146 6034 raft_consensus.cc:2454] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1dc6c17624d4369ad0fad1373a1bea3 in term 1.
I20260805 23:05:09.680490 5710 leader_election.cc:302] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbc802857f34ca794d76681f52dd582, b1dc6c17624d4369ad0fad1373a1bea3; no voters:
I20260805 23:05:09.680716 6525 raft_consensus.cc:2781] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:09.680967 6525 raft_consensus.cc:686] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [term 1 LEADER]: Becoming Leader. State: Replica: b1dc6c17624d4369ad0fad1373a1bea3, State: Running, Role: LEADER
I20260805 23:05:09.681133 6525 consensus_queue.cc:237] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } }
I20260805 23:05:09.683291 5574 catalog_manager.cc:5495] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65). New cstate: current_term: 1 leader_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1dc6c17624d4369ad0fad1373a1bea3" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 45319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:09.726683 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:09.723371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.3.12.65:45319)
0805 23:05:09.723743 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:09.723746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:09.723756 (+ 10us) server_negotiation.cc:407] Connection header received
0805 23:05:09.723792 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:09.723794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:09.723832 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:09.723889 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:09.724179 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.724773 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.725650 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:09.725862 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:09.725944 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:09.726204 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:09.726272 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:09.726331 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:09.726439 (+ 108us) server_negotiation.cc:299] Negotiation successful
0805 23:05:09.726485 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260805 23:05:09.733999 6255 raft_consensus.cc:1261] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Refusing update from remote peer b1dc6c17624d4369ad0fad1373a1bea3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:09.734153 6034 raft_consensus.cc:1261] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Refusing update from remote peer b1dc6c17624d4369ad0fad1373a1bea3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:09.734606 6525 consensus_queue.cc:1035] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8aef2324cfdb4b2288b8798993eb5030" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:09.734751 6557 consensus_queue.cc:1035] T 7a39d11c72904d9bb90ef86603ca766c P b1dc6c17624d4369ad0fad1373a1bea3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbc802857f34ca794d76681f52dd582" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 23:05:09.749238 6470 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260805 23:05:11.301432 5574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:11.301635 5574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:11.304155 5574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16c25aca341845d4bc8aa12d08c1a98c: Sending DeleteTablet for 3 replicas of tablet 7a39d11c72904d9bb90ef86603ca766c
I20260805 23:05:11.304761 5788 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a39d11c72904d9bb90ef86603ca766c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:11 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:57264
I20260805 23:05:11.304952 6217 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a39d11c72904d9bb90ef86603ca766c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:11 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:54384
I20260805 23:05:11.305059 5996 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a39d11c72904d9bb90ef86603ca766c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:11 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:40660
I20260805 23:05:11.305323 6598 tablet_replica.cc:330] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: stopping tablet replica
I20260805 23:05:11.305454 6600 tablet_replica.cc:330] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: stopping tablet replica
I20260805 23:05:11.305533 6598 raft_consensus.cc:2229] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:11.305756 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 5684
I20260805 23:05:11.305791 6598 raft_consensus.cc:2258] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:11.323127 6600 raft_consensus.cc:2229] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:11.323333 6600 raft_consensus.cc:2258] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:11.323598 6598 ts_tablet_manager.cc:1905] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:11.324028 6600 ts_tablet_manager.cc:1905] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:11.326692 6598 ts_tablet_manager.cc:1918] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 23:05:11.326773 6600 ts_tablet_manager.cc:1918] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260805 23:05:11.326856 6600 log.cc:1196] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/7a39d11c72904d9bb90ef86603ca766c
I20260805 23:05:11.326951 6598 log.cc:1196] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/7a39d11c72904d9bb90ef86603ca766c
I20260805 23:05:11.327171 6600 ts_tablet_manager.cc:1939] T 7a39d11c72904d9bb90ef86603ca766c P 8aef2324cfdb4b2288b8798993eb5030: Deleting consensus metadata
I20260805 23:05:11.327325 6598 ts_tablet_manager.cc:1939] T 7a39d11c72904d9bb90ef86603ca766c P 0bbc802857f34ca794d76681f52dd582: Deleting consensus metadata
I20260805 23:05:11.328105 5555 catalog_manager.cc:4841] TS 0bbc802857f34ca794d76681f52dd582 (127.3.12.67:43535): tablet 7a39d11c72904d9bb90ef86603ca766c (table test-table [id=01492e1aa6d944e7b8cd040b3c4aaf68]) successfully deleted
I20260805 23:05:11.328106 5554 catalog_manager.cc:4841] TS 8aef2324cfdb4b2288b8798993eb5030 (127.3.12.66:35041): tablet 7a39d11c72904d9bb90ef86603ca766c (table test-table [id=01492e1aa6d944e7b8cd040b3c4aaf68]) successfully deleted
W20260805 23:05:11.329452 5557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:45319 (error 108)
I20260805 23:05:11.329597 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 5897
I20260805 23:05:11.332841 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:11.332320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:45319 (local address 127.0.0.1:57308)
0805 23:05:11.332683 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:11.332752 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
W20260805 23:05:11.333119 5557 catalog_manager.cc:4584] TS b1dc6c17624d4369ad0fad1373a1bea3 (127.3.12.65:45319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a39d11c72904d9bb90ef86603ca766c: Network error: Client connection negotiation failed: client connection to 127.3.12.65:45319: connect: Connection refused (error 111)
I20260805 23:05:11.342437 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 6124
I20260805 23:05:11.356540 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 5528
2026-08-05T23:05:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6968 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:11 dist-test-slave-2d19 krb5kdc[6607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:11 dist-test-slave-2d19 krb5kdc[6607](info): set up 2 sockets
Aug 05 23:05:11 dist-test-slave-2d19 krb5kdc[6607](info): commencing operation
krb5kdc: starting...
W20260805 23:05:12.737195 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.352s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260805 23:05:12.880784 3121 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.143s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260805 23:05:13.083843 3121 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.203s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260805 23:05:13.265239 3121 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.181s user 0.000s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/test-user.keytab.
W20260805 23:05:13.922722 3121 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.657s user 0.000s sys 0.002s
Aug 05 23:05:13 dist-test-slave-2d19 krb5kdc[6607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
W20260805 23:05:14.157227 3121 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.12.126: real 0.209s user 0.000s sys 0.004s
I20260805 23:05:14.157783 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:38413
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:38413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:14.273684 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:14.274032 6628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:14.274080 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:14.278189 6628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:14.278257 6628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:14.278277 6628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:14.278294 6628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:14.278311 6628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:14.283452 6628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:38413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:14.284613 6628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:14.285604 6628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:14.294476 6635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:14.294469 6638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:14.294469 6640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:14.294624 6628 server_base.cc:1034] running on GCE node
I20260805 23:05:14.295451 6628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:14.296641 6628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:14.297847 6628 hybrid_clock.cc:648] HybridClock initialized: now 1785971114297830 us; error 28 us; skew 500 ppm
Aug 05 23:05:14 dist-test-slave-2d19 krb5kdc[6607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:14.300802 6628 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:14.301826 6628 webserver.cc:460] Webserver started at http://127.3.12.126:39803/ using document root <none> and password file <none>
I20260805 23:05:14.302404 6628 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:14.302503 6628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:14.302793 6628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:14.304947 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "a191bb25a222457494c4b8204a209711"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "7537fb640af10eacb1c00b515c489828"
server_key_iv: "843c962c9f3f5b25307c979204f4cac6"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.305533 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "a191bb25a222457494c4b8204a209711"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "7537fb640af10eacb1c00b515c489828"
server_key_iv: "843c962c9f3f5b25307c979204f4cac6"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.309355 6628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:14.311805 6654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:14.313108 6628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:14.313272 6628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "a191bb25a222457494c4b8204a209711"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "7537fb640af10eacb1c00b515c489828"
server_key_iv: "843c962c9f3f5b25307c979204f4cac6"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.313387 6628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:14.325484 6628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:14.329010 6628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:14.329236 6628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:14.336668 6628 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:38413
I20260805 23:05:14.336689 6742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:38413 every 8 connection(s)
I20260805 23:05:14.337674 6628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:14.340551 6743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:14.345605 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 6628
I20260805 23:05:14.345741 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:14.346024 3121 external_mini_cluster.cc:1428] Setting key 5f1dd14e20db24869bea217b7662b202
I20260805 23:05:14.346740 6743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711: Bootstrap starting.
I20260805 23:05:14.349496 6743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:14.350544 6743 log.cc:824] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:14.352968 6743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711: No bootstrap required, opened a new log
Aug 05 23:05:14 dist-test-slave-2d19 krb5kdc[6607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:14.357479 6743 raft_consensus.cc:348] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a191bb25a222457494c4b8204a209711" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38413 } }
I20260805 23:05:14.357749 6743 raft_consensus.cc:374] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:14.357828 6743 raft_consensus.cc:729] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a191bb25a222457494c4b8204a209711, State: Initialized, Role: FOLLOWER
I20260805 23:05:14.358378 6743 consensus_queue.cc:260] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a191bb25a222457494c4b8204a209711" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38413 } }
I20260805 23:05:14.358561 6743 raft_consensus.cc:388] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:14.358649 6743 raft_consensus.cc:482] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:14.358763 6743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:14.360030 6743 raft_consensus.cc:504] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a191bb25a222457494c4b8204a209711" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38413 } }
I20260805 23:05:14.360436 6743 leader_election.cc:302] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a191bb25a222457494c4b8204a209711; no voters:
I20260805 23:05:14.360817 6743 leader_election.cc:288] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:14.360922 6749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:14.361235 6749 raft_consensus.cc:686] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [term 1 LEADER]: Becoming Leader. State: Replica: a191bb25a222457494c4b8204a209711, State: Running, Role: LEADER
I20260805 23:05:14.361624 6749 consensus_queue.cc:237] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a191bb25a222457494c4b8204a209711" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38413 } }
I20260805 23:05:14.362195 6743 sys_catalog.cc:564] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:14.362550 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:14.347689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.3.12.126:38413)
0805 23:05:14.348199 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:14.348216 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:14.348307 (+ 91us) server_negotiation.cc:407] Connection header received
0805 23:05:14.349070 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:14.349090 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:14.349438 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:14.349800 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:14.350751 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.352032 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.353218 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.353679 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.353799 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:14.356707 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:14.356712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:14.356726 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:14.356755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:14.358902 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:14.359436 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:14.359443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:14.359448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:14.359574 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:14.359896 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:14.359899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:14.359900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:14.360446 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:14.360666 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:14.361002 (+ 336us) server_negotiation.cc:299] Negotiation successful
0805 23:05:14.361531 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":156,"threads_started":1}
I20260805 23:05:14.362866 6749 sys_catalog.cc:455] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a191bb25a222457494c4b8204a209711. Latest consensus state: current_term: 1 leader_uuid: "a191bb25a222457494c4b8204a209711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a191bb25a222457494c4b8204a209711" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38413 } } }
I20260805 23:05:14.362996 6749 sys_catalog.cc:458] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:14.363545 6752 sys_catalog.cc:455] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a191bb25a222457494c4b8204a209711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a191bb25a222457494c4b8204a209711" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38413 } } }
I20260805 23:05:14.363756 6752 sys_catalog.cc:458] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:14.364946 6761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:14.368089 6761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:14.374531 6761 catalog_manager.cc:1347] Generated new cluster ID: 3a9da50830b54398905617738567b9e8
I20260805 23:05:14.374637 6761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:14.388337 6761 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:14.389339 6761 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:14.400067 6761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a191bb25a222457494c4b8204a209711: Generated new TSK 0
I20260805 23:05:14.400872 6761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:05:14.438798 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38413
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:14.578511 6779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:14.578778 6779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:14.578820 6779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:14.582731 6779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:14.582785 6779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:14.582844 6779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:14.588125 6779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:14.589224 6779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:14.590219 6779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:14.599985 6789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:14.599947 6794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:14.599972 6790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:14.600240 6779 server_base.cc:1034] running on GCE node
I20260805 23:05:14.600708 6779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:14.601425 6779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:14.602592 6779 hybrid_clock.cc:648] HybridClock initialized: now 1785971114602572 us; error 31 us; skew 500 ppm
Aug 05 23:05:14 dist-test-slave-2d19 krb5kdc[6607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:14.605860 6779 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:05:14.606943 6779 webserver.cc:460] Webserver started at http://127.3.12.65:41221/ using document root <none> and password file <none>
I20260805 23:05:14.607515 6779 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:14.607583 6779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:14.607820 6779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:14.609951 6779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "4a97e233160b4733a662529b207d834f"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "3e14daa83ef2bb8794b688475002e9f4"
server_key_iv: "74f31fdf2397854d85e4ef55fdaadc78"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.610541 6779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "4a97e233160b4733a662529b207d834f"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "3e14daa83ef2bb8794b688475002e9f4"
server_key_iv: "74f31fdf2397854d85e4ef55fdaadc78"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.614391 6779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260805 23:05:14.616976 6804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:14.618350 6779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:14.618492 6779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "4a97e233160b4733a662529b207d834f"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "3e14daa83ef2bb8794b688475002e9f4"
server_key_iv: "74f31fdf2397854d85e4ef55fdaadc78"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.618608 6779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:14.628916 6779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:14.632200 6779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:14.632427 6779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:14.633023 6779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:14.633855 6779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:14.633924 6779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:14.633999 6779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:14.634039 6779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:14.643661 6779 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:39075
I20260805 23:05:14.643680 6966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:39075 every 8 connection(s)
I20260805 23:05:14.644716 6779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:14.647675 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 6779
I20260805 23:05:14.647802 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:14.648095 3121 external_mini_cluster.cc:1428] Setting key 143ef08214d891adbe9ca26d7a28c3de
Aug 05 23:05:14 dist-test-slave-2d19 krb5kdc[6607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:14.660454 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:14.646622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:53409 (local address 127.3.12.126:38413)
0805 23:05:14.646774 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:14.646778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:14.647687 (+ 909us) server_negotiation.cc:407] Connection header received
0805 23:05:14.649386 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:14.649389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:14.649458 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:14.649552 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:14.651402 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.651984 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.653276 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.653514 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.653632 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:14.656592 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:14.656595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:14.656597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:14.656622 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:14.658263 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:14.658838 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:14.658841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:14.658843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:14.658920 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:14.659277 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:14.659280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:14.659282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:14.659470 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:14.659576 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:14.660111 (+ 535us) server_negotiation.cc:299] Negotiation successful
0805 23:05:14.660274 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 23:05:14.661599 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:14.647003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38413 (local address 127.3.12.65:53409)
0805 23:05:14.647503 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:14.647550 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:14.649166 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:14.649822 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:14.649833 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:14.650412 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:14.651209 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:14.651234 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.652153 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.652156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:14.653134 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:14.653152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.653683 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.653687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:14.653862 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:14.654542 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:14.654559 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:14.656410 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:14.658391 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:14.658398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:14.658414 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:14.658723 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:14.659018 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:14.659021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:14.659023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:14.659172 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:14.659577 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:14.659585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:14.659860 (+ 275us) client_negotiation.cc:769] Sending connection context
0805 23:05:14.660057 (+ 197us) client_negotiation.cc:240] Negotiation successful
0805 23:05:14.660441 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:05:14.663149 6969 heartbeater.cc:347] Connected to a master server at 127.3.12.126:38413
I20260805 23:05:14.663453 6969 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:14.664129 6969 heartbeater.cc:511] Master 127.3.12.126:38413 requested a full tablet report, sending...
I20260805 23:05:14.665771 6690 ts_manager.cc:194] Registered new tserver with Master: 4a97e233160b4733a662529b207d834f (127.3.12.65:39075)
I20260805 23:05:14.667506 6690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:53409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:14.681432 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38413
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:14.799891 6975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:14.800160 6975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:14.800206 6975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:14.804383 6975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:14.804440 6975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:14.804499 6975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:14.809616 6975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:14.810622 6975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:14.811599 6975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:14.821278 6984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:14.821388 6975 server_base.cc:1034] running on GCE node
W20260805 23:05:14.821229 6987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:14.821229 6985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:14.821972 6975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:14.822654 6975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:14.823860 6975 hybrid_clock.cc:648] HybridClock initialized: now 1785971114823841 us; error 30 us; skew 500 ppm
Aug 05 23:05:14 dist-test-slave-2d19 krb5kdc[6607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:14.826731 6975 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:14.827621 6975 webserver.cc:460] Webserver started at http://127.3.12.66:41735/ using document root <none> and password file <none>
I20260805 23:05:14.828217 6975 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:14.828379 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:14.828610 6975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:14.830677 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "8c48de67a8a4447ead65fb5dca2b6026"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "4b4be3feb1ca936843daf4c554160182"
server_key_iv: "ead1a23d2cf0458bcd72962885325845"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.831261 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "8c48de67a8a4447ead65fb5dca2b6026"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "4b4be3feb1ca936843daf4c554160182"
server_key_iv: "ead1a23d2cf0458bcd72962885325845"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.835222 6975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260805 23:05:14.837802 6994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:14.839082 6975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:14.839241 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "8c48de67a8a4447ead65fb5dca2b6026"
format_stamp: "Formatted at 2026-08-05 23:05:14 on dist-test-slave-2d19"
server_key: "4b4be3feb1ca936843daf4c554160182"
server_key_iv: "ead1a23d2cf0458bcd72962885325845"
server_key_version: "encryptionkey@0"
I20260805 23:05:14.839357 6975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:14.870976 6975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:14.874308 6975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:14.874521 6975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:14.875152 6975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:14.876116 6975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:14.876184 6975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:14.876302 6975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:14.876355 6975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:14.885792 6975 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:43839
I20260805 23:05:14.885840 7161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:43839 every 8 connection(s)
I20260805 23:05:14.886785 6975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:14.889559 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 6975
I20260805 23:05:14.889669 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:14.889957 3121 external_mini_cluster.cc:1428] Setting key 6161c9d49be0b94269f0deef7e3c2ba8
Aug 05 23:05:14 dist-test-slave-2d19 krb5kdc[6607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971114, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:05:14.903070 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:14.888806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:55995 (local address 127.3.12.126:38413)
0805 23:05:14.888948 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:14.888952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:14.889916 (+ 964us) server_negotiation.cc:407] Connection header received
0805 23:05:14.891679 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:14.891681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:14.891732 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:14.891823 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:14.893135 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.893914 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.895071 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.895454 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.895578 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:14.898496 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:14.898502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:14.898508 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:14.898544 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:14.900453 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:14.901091 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:14.901096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:14.901100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:14.901195 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:14.901692 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:14.901700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:14.901702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:14.901898 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:14.902022 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:14.902559 (+ 537us) server_negotiation.cc:299] Negotiation successful
0805 23:05:14.902845 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:05:14.903980 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:14.889229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38413 (local address 127.3.12.66:55995)
0805 23:05:14.889738 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:14.889794 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:14.891441 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:14.891994 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:14.892001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:14.892480 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:14.892989 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:14.893003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.894051 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.894054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:14.894940 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:14.894949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:14.895639 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:14.895645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:14.895810 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:14.896483 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:14.896501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:14.898303 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:14.900597 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:14.900604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:14.900618 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:14.900961 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:14.901322 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:14.901325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:14.901326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:14.901509 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:14.902022 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:14.902030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:14.902302 (+ 272us) client_negotiation.cc:769] Sending connection context
0805 23:05:14.902515 (+ 213us) client_negotiation.cc:240] Negotiation successful
0805 23:05:14.902889 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
I20260805 23:05:14.905308 7163 heartbeater.cc:347] Connected to a master server at 127.3.12.126:38413
I20260805 23:05:14.905589 7163 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:14.906131 7163 heartbeater.cc:511] Master 127.3.12.126:38413 requested a full tablet report, sending...
I20260805 23:05:14.907275 6690 ts_manager.cc:194] Registered new tserver with Master: 8c48de67a8a4447ead65fb5dca2b6026 (127.3.12.66:43839)
I20260805 23:05:14.907871 6690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:55995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:14.923593 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38413
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:15.046836 7170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:15.047137 7170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:15.047189 7170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:15.051781 7170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:15.051851 7170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:15.051932 7170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:15.057629 7170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:15.058687 7170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:15.059689 7170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:15.070076 7184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:15.070128 7170 server_base.cc:1034] running on GCE node
W20260805 23:05:15.070155 7178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:15.070076 7181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:15.070801 7170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:15.071496 7170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:15.072685 7170 hybrid_clock.cc:648] HybridClock initialized: now 1785971115072667 us; error 30 us; skew 500 ppm
Aug 05 23:05:15 dist-test-slave-2d19 krb5kdc[6607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971115, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:15.075495 7170 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:15.076467 7170 webserver.cc:460] Webserver started at http://127.3.12.67:39079/ using document root <none> and password file <none>
I20260805 23:05:15.076996 7170 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:15.077054 7170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:15.077234 7170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:15.079262 7170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "db8dc5e1376c41109de5426627c32c00"
format_stamp: "Formatted at 2026-08-05 23:05:15 on dist-test-slave-2d19"
server_key: "4982adc5726b5aa5b892854b636c6f7e"
server_key_iv: "fd49342753f017dbf2a82dc8c8007011"
server_key_version: "encryptionkey@0"
I20260805 23:05:15.079797 7170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "db8dc5e1376c41109de5426627c32c00"
format_stamp: "Formatted at 2026-08-05 23:05:15 on dist-test-slave-2d19"
server_key: "4982adc5726b5aa5b892854b636c6f7e"
server_key_iv: "fd49342753f017dbf2a82dc8c8007011"
server_key_version: "encryptionkey@0"
I20260805 23:05:15.083694 7170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:05:15.086426 7193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:15.087774 7170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:15.087908 7170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "db8dc5e1376c41109de5426627c32c00"
format_stamp: "Formatted at 2026-08-05 23:05:15 on dist-test-slave-2d19"
server_key: "4982adc5726b5aa5b892854b636c6f7e"
server_key_iv: "fd49342753f017dbf2a82dc8c8007011"
server_key_version: "encryptionkey@0"
I20260805 23:05:15.088027 7170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:15.120913 7170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:15.124243 7170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:15.124495 7170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:15.125120 7170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:15.126008 7170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:15.126070 7170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:15.126145 7170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:15.126183 7170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:15.135433 7170 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:39889
I20260805 23:05:15.135455 7371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:39889 every 8 connection(s)
I20260805 23:05:15.136495 7170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:15.141896 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 7170
I20260805 23:05:15.142019 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:15.142342 3121 external_mini_cluster.cc:1428] Setting key 63a887ef5841708f92b8af6149464554
Aug 05 23:05:15 dist-test-slave-2d19 krb5kdc[6607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971115, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:15.151027 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:15.138488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:42757 (local address 127.3.12.126:38413)
0805 23:05:15.138626 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:15.138630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:15.139424 (+ 794us) server_negotiation.cc:407] Connection header received
0805 23:05:15.140865 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:15.140869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:15.140929 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:15.141032 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:15.142518 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:15.143301 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:15.144911 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:15.145131 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:15.145215 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:15.147685 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:15.147688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:15.147691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:15.147715 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:15.149242 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:15.149707 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:15.149710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:15.149711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:15.149786 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:15.150077 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:15.150079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:15.150080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:15.150251 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:15.150340 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:15.150800 (+ 460us) server_negotiation.cc:299] Negotiation successful
0805 23:05:15.150903 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:05:15.151882 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:15.138822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38413 (local address 127.3.12.67:42757)
0805 23:05:15.139268 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:15.139313 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:15.140645 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:15.141209 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:15.141216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:15.141572 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:15.142311 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:15.142332 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:15.143437 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:15.143441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:15.144781 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:15.144792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:15.145263 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:15.145265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:15.145406 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:15.146025 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:15.146041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:15.147529 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:15.149343 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:15.149348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:15.149360 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:15.149620 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:15.149863 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:15.149865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:15.149866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:15.150001 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:15.150338 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:15.150343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:15.150581 (+ 238us) client_negotiation.cc:769] Sending connection context
0805 23:05:15.150752 (+ 171us) client_negotiation.cc:240] Negotiation successful
0805 23:05:15.151016 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260805 23:05:15.153350 7373 heartbeater.cc:347] Connected to a master server at 127.3.12.126:38413
I20260805 23:05:15.153689 7373 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:15.154362 7373 heartbeater.cc:511] Master 127.3.12.126:38413 requested a full tablet report, sending...
I20260805 23:05:15.155624 6690 ts_manager.cc:194] Registered new tserver with Master: db8dc5e1376c41109de5426627c32c00 (127.3.12.67:39889)
I20260805 23:05:15.156308 6690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:42757
I20260805 23:05:15.156608 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260805 23:05:15.171213 6747 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:15.169676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.3.12.126:38413)
0805 23:05:15.169841 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:15.169846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:15.170006 (+ 160us) server_negotiation.cc:407] Connection header received
0805 23:05:15.170184 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:15.170187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:15.170236 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:15.170320 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:15.171059 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39650: BlockingRecv error: recv got EOF from 127.0.0.1:39650 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:05:15.172541 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 6779
I20260805 23:05:15.196563 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 6975
I20260805 23:05:15.215828 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 7170
I20260805 23:05:15.241969 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 6628
2026-08-05T23:05:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3884 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:15 dist-test-slave-2d19 krb5kdc[7387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:15 dist-test-slave-2d19 krb5kdc[7387](info): set up 2 sockets
Aug 05 23:05:15 dist-test-slave-2d19 krb5kdc[7387](info): commencing operation
krb5kdc: starting...
W20260805 23:05:16.474423 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:16 dist-test-slave-2d19 krb5kdc[7387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:16.623281 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36525
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:36525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:16.736179 7408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:16.736464 7408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:16.736503 7408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:16.740381 7408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:16.740429 7408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:16.740448 7408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:16.740464 7408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:16.740479 7408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:16.746022 7408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:36525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:16.747035 7408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:16.748462 7408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:16.756781 7419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:16.756942 7408 server_base.cc:1034] running on GCE node
W20260805 23:05:16.756789 7417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:16.757121 7416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:16.757704 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:16.758800 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:16.759989 7408 hybrid_clock.cc:648] HybridClock initialized: now 1785971116759980 us; error 30 us; skew 500 ppm
Aug 05 23:05:16 dist-test-slave-2d19 krb5kdc[7387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971116, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:16.762734 7408 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:16.763608 7408 webserver.cc:460] Webserver started at http://127.3.12.126:43077/ using document root <none> and password file <none>
I20260805 23:05:16.764117 7408 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:16.764185 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:16.764418 7408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:16.766398 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "524dcb18aa4e467386599e8eeac07006"
format_stamp: "Formatted at 2026-08-05 23:05:16 on dist-test-slave-2d19"
server_key: "4e346d2f54d0d50765dfd93ae5e2cd72"
server_key_iv: "5b30ef80585a009b71450546240e55f9"
server_key_version: "encryptionkey@0"
I20260805 23:05:16.766901 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "524dcb18aa4e467386599e8eeac07006"
format_stamp: "Formatted at 2026-08-05 23:05:16 on dist-test-slave-2d19"
server_key: "4e346d2f54d0d50765dfd93ae5e2cd72"
server_key_iv: "5b30ef80585a009b71450546240e55f9"
server_key_version: "encryptionkey@0"
I20260805 23:05:16.770624 7408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:05:16.773053 7427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:16.774264 7408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:16.774392 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "524dcb18aa4e467386599e8eeac07006"
format_stamp: "Formatted at 2026-08-05 23:05:16 on dist-test-slave-2d19"
server_key: "4e346d2f54d0d50765dfd93ae5e2cd72"
server_key_iv: "5b30ef80585a009b71450546240e55f9"
server_key_version: "encryptionkey@0"
I20260805 23:05:16.774503 7408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:16.795528 7408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:16.798879 7408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:16.799093 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:16.806478 7408 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:36525
I20260805 23:05:16.806808 7507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:36525 every 8 connection(s)
I20260805 23:05:16.807667 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:16.810475 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 7408
I20260805 23:05:16.810602 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:16.810611 7509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:16.810914 3121 external_mini_cluster.cc:1428] Setting key 641e47057efaff2d4ff5f310cfc8e758
I20260805 23:05:16.817142 7509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006: Bootstrap starting.
I20260805 23:05:16.819696 7509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006: Neither blocks nor log segments found. Creating new log.
Aug 05 23:05:16 dist-test-slave-2d19 krb5kdc[7387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:16.820518 7509 log.cc:824] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:16.822938 7509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006: No bootstrap required, opened a new log
I20260805 23:05:16.826011 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:16.812414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.3.12.126:36525)
0805 23:05:16.812870 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:16.812886 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:16.812921 (+ 35us) server_negotiation.cc:407] Connection header received
0805 23:05:16.813832 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:16.813851 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:16.814181 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:16.814515 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:16.815374 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:16.816406 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:16.817304 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:16.817970 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:16.818149 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:16.820686 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:16.820692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:16.820709 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:16.820739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:16.822757 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:16.823226 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:16.823232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:16.823237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:16.823389 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:16.823660 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:16.823663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:16.823665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:16.824179 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:16.824397 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:16.824619 (+ 222us) server_negotiation.cc:299] Negotiation successful
0805 23:05:16.824930 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260805 23:05:16.827620 7509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524dcb18aa4e467386599e8eeac07006" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36525 } }
I20260805 23:05:16.827888 7509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:16.828009 7509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524dcb18aa4e467386599e8eeac07006, State: Initialized, Role: FOLLOWER
I20260805 23:05:16.828519 7509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524dcb18aa4e467386599e8eeac07006" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36525 } }
I20260805 23:05:16.828672 7509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:16.828752 7509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:16.828847 7509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:16.829036 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:05:16.829855 7509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524dcb18aa4e467386599e8eeac07006" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36525 } }
I20260805 23:05:16.830194 7509 leader_election.cc:302] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 524dcb18aa4e467386599e8eeac07006; no voters:
I20260805 23:05:16.830514 7509 leader_election.cc:288] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:16.830654 7516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:16.830883 7516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [term 1 LEADER]: Becoming Leader. State: Replica: 524dcb18aa4e467386599e8eeac07006, State: Running, Role: LEADER
I20260805 23:05:16.831283 7516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524dcb18aa4e467386599e8eeac07006" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36525 } }
I20260805 23:05:16.831948 7509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:16.833216 7518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 524dcb18aa4e467386599e8eeac07006. Latest consensus state: current_term: 1 leader_uuid: "524dcb18aa4e467386599e8eeac07006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524dcb18aa4e467386599e8eeac07006" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36525 } } }
I20260805 23:05:16.833370 7518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:16.833215 7517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "524dcb18aa4e467386599e8eeac07006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524dcb18aa4e467386599e8eeac07006" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36525 } } }
I20260805 23:05:16.833711 7517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:16.833770 7534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:16.837117 7534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:16.842319 7534 catalog_manager.cc:1347] Generated new cluster ID: c02f7e0d4e444b63bde6bdabeee0945f
I20260805 23:05:16.842381 7534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:16.854322 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:16.848515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.3.12.126:36525)
0805 23:05:16.848630 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:16.848635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:16.848911 (+ 276us) server_negotiation.cc:407] Connection header received
0805 23:05:16.849030 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:16.849034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:16.849092 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:16.849185 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:16.849744 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:16.850571 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:16.851738 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:16.852124 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:16.852234 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:16.853175 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:16.853179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:16.853182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:16.853818 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:16.853856 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:16.853876 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:16.853972 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:16.854093 (+ 121us) server_negotiation.cc:299] Negotiation successful
0805 23:05:16.854163 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260805 23:05:16.856246 7534 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:16.857471 7534 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:16.871440 7534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 524dcb18aa4e467386599e8eeac07006: Generated new TSK 0
I20260805 23:05:16.872586 7534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:05:16.886323 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 7408
2026-08-05T23:05:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1645 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:16 dist-test-slave-2d19 krb5kdc[7569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:16 dist-test-slave-2d19 krb5kdc[7569](info): set up 2 sockets
Aug 05 23:05:16 dist-test-slave-2d19 krb5kdc[7569](info): commencing operation
krb5kdc: starting...
W20260805 23:05:18.118551 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:18 dist-test-slave-2d19 krb5kdc[7569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:18.255436 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:32809
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:32809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:18.370329 7594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:18.370664 7594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:18.370741 7594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:18.374879 7594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:18.374977 7594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:18.375012 7594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:18.375057 7594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:18.375089 7594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:18.380460 7594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:32809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:32809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:18.381560 7594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:18.382624 7594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:18.391260 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:18.391422 7594 server_base.cc:1034] running on GCE node
W20260805 23:05:18.391255 7603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:18.391255 7600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:18.392169 7594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:18.393379 7594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:18.394572 7594 hybrid_clock.cc:648] HybridClock initialized: now 1785971118394552 us; error 32 us; skew 500 ppm
Aug 05 23:05:18 dist-test-slave-2d19 krb5kdc[7569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:18.397542 7594 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:18.398459 7594 webserver.cc:460] Webserver started at http://127.3.12.126:42993/ using document root <none> and password file <none>
I20260805 23:05:18.398983 7594 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:18.399044 7594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:18.399226 7594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:18.401314 7594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "08640b7226f24f61b35848e3ef986bfe"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "9072236cb9f6f729ab149eacc26cdd2e"
server_key_iv: "83b2d94208bc4a6d7e3413b2c6f24f39"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.401834 7594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "08640b7226f24f61b35848e3ef986bfe"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "9072236cb9f6f729ab149eacc26cdd2e"
server_key_iv: "83b2d94208bc4a6d7e3413b2c6f24f39"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.405685 7594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:18.408046 7615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:18.409351 7594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:18.409456 7594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "08640b7226f24f61b35848e3ef986bfe"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "9072236cb9f6f729ab149eacc26cdd2e"
server_key_iv: "83b2d94208bc4a6d7e3413b2c6f24f39"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.409544 7594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:18.435177 7594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:18.438659 7594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:18.438823 7594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:18.446192 7594 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:32809
I20260805 23:05:18.446558 7688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:32809 every 8 connection(s)
I20260805 23:05:18.447232 7594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:18.450146 7694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:18.453126 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 7594
I20260805 23:05:18.453260 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:18.453578 3121 external_mini_cluster.cc:1428] Setting key ba58094693dcdd03813eb486e846f704
I20260805 23:05:18.456604 7694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe: Bootstrap starting.
I20260805 23:05:18.459267 7694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:18.460143 7694 log.cc:824] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:18.462283 7694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe: No bootstrap required, opened a new log
Aug 05 23:05:18 dist-test-slave-2d19 krb5kdc[7569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:18.466898 7694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08640b7226f24f61b35848e3ef986bfe" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 32809 } }
I20260805 23:05:18.467216 7694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:18.467330 7694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08640b7226f24f61b35848e3ef986bfe, State: Initialized, Role: FOLLOWER
I20260805 23:05:18.467782 7694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08640b7226f24f61b35848e3ef986bfe" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 32809 } }
I20260805 23:05:18.467906 7694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:18.468010 7694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:18.468112 7694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:18.468603 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:18.455026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.3.12.126:32809)
0805 23:05:18.455455 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:18.455472 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:18.455510 (+ 38us) server_negotiation.cc:407] Connection header received
0805 23:05:18.456132 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:18.456149 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:18.456566 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:18.456893 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:18.457771 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:18.458762 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:18.460004 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:18.460556 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:18.460733 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:18.463550 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:18.463555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:18.463568 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:18.463596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:18.465385 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:18.465885 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:18.465890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:18.465894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:18.466005 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:18.466324 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:18.466326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:18.466327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:18.466706 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:18.466855 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:18.467213 (+ 358us) server_negotiation.cc:299] Negotiation successful
0805 23:05:18.467590 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
I20260805 23:05:18.469169 7694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08640b7226f24f61b35848e3ef986bfe" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 32809 } }
I20260805 23:05:18.469528 7694 leader_election.cc:302] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08640b7226f24f61b35848e3ef986bfe; no voters:
I20260805 23:05:18.469863 7694 leader_election.cc:288] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:18.469945 7699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:18.470155 7699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [term 1 LEADER]: Becoming Leader. State: Replica: 08640b7226f24f61b35848e3ef986bfe, State: Running, Role: LEADER
I20260805 23:05:18.470535 7699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08640b7226f24f61b35848e3ef986bfe" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 32809 } }
I20260805 23:05:18.471225 7694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:18.472380 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08640b7226f24f61b35848e3ef986bfe. Latest consensus state: current_term: 1 leader_uuid: "08640b7226f24f61b35848e3ef986bfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08640b7226f24f61b35848e3ef986bfe" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 32809 } } }
I20260805 23:05:18.472513 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:18.472880 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08640b7226f24f61b35848e3ef986bfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08640b7226f24f61b35848e3ef986bfe" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 32809 } } }
I20260805 23:05:18.472939 7714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:18.473035 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:18.476217 7714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:18.482411 7714 catalog_manager.cc:1347] Generated new cluster ID: 0c6824f7ecf6402dbf11f789f0536f77
I20260805 23:05:18.482502 7714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:18.499320 7714 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:18.500701 7714 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:18.517709 7714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08640b7226f24f61b35848e3ef986bfe: Generated new TSK 0
I20260805 23:05:18.518733 7714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:05:18.551513 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:32809
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:18.668596 7733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:18.668892 7733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:18.668977 7733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:18.673317 7733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:18.673394 7733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:18.673482 7733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:18.679064 7733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:32809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:18.680313 7733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:18.681420 7733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:18.691514 7741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:18.691525 7742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:18.691792 7744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:18.692013 7733 server_base.cc:1034] running on GCE node
I20260805 23:05:18.692467 7733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:18.693085 7733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:18.694250 7733 hybrid_clock.cc:648] HybridClock initialized: now 1785971118694217 us; error 44 us; skew 500 ppm
Aug 05 23:05:18 dist-test-slave-2d19 krb5kdc[7569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:18.697209 7733 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:05:18.698052 7733 webserver.cc:460] Webserver started at http://127.3.12.65:46773/ using document root <none> and password file <none>
I20260805 23:05:18.698639 7733 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:18.698704 7733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:18.698874 7733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:18.700968 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "8d9839cd07b14087b1dfb49def5ae2b5"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "f03ead640ecf9b00045b27b9dafa1d25"
server_key_iv: "dd5769c1fbc7f7403fdaf23f99e65c59"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.701488 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "8d9839cd07b14087b1dfb49def5ae2b5"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "f03ead640ecf9b00045b27b9dafa1d25"
server_key_iv: "dd5769c1fbc7f7403fdaf23f99e65c59"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.705171 7733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260805 23:05:18.707511 7752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:18.709007 7733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:18.709149 7733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "8d9839cd07b14087b1dfb49def5ae2b5"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "f03ead640ecf9b00045b27b9dafa1d25"
server_key_iv: "dd5769c1fbc7f7403fdaf23f99e65c59"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.709270 7733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:18.722199 7733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:18.725490 7733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:18.725706 7733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:18.726320 7733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:18.727152 7733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:18.727221 7733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:18.727289 7733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:18.727339 7733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:18.736400 7733 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:37615
I20260805 23:05:18.736421 7935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:37615 every 8 connection(s)
I20260805 23:05:18.737404 7733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:18.739431 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 7733
I20260805 23:05:18.739559 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:18.739873 3121 external_mini_cluster.cc:1428] Setting key da14874e24e5b12a2e710d93f0d0370f
Aug 05 23:05:18 dist-test-slave-2d19 krb5kdc[7569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:18.752841 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:18.739382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:43871 (local address 127.3.12.126:32809)
0805 23:05:18.739554 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:18.739562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:18.740711 (+ 1149us) server_negotiation.cc:407] Connection header received
0805 23:05:18.742649 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:18.742651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:18.742702 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:18.742767 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:18.744442 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:18.744987 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:18.746161 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:18.746392 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:18.746502 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:18.749115 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:18.749118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:18.749120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:18.749145 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:18.750671 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:18.751198 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:18.751201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:18.751203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:18.751290 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:18.751747 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:18.751750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:18.751751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:18.751958 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:18.752044 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:18.752604 (+ 560us) server_negotiation.cc:299] Negotiation successful
0805 23:05:18.752711 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:05:18.753945 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:18.739897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:32809 (local address 127.3.12.65:43871)
0805 23:05:18.740507 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:18.740585 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:18.742363 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:18.742992 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:18.743004 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:18.743598 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:18.744234 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:18.744300 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:18.745129 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:18.745132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:18.746024 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:18.746035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:18.746557 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:18.746562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:18.746730 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:18.747388 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:18.747405 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:18.748935 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:18.750793 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:18.750799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:18.750816 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:18.751089 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:18.751386 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:18.751389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:18.751391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:18.751627 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:18.752063 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:18.752070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:18.752377 (+ 307us) client_negotiation.cc:769] Sending connection context
0805 23:05:18.752544 (+ 167us) client_negotiation.cc:240] Negotiation successful
0805 23:05:18.752910 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":164,"threads_started":1}
I20260805 23:05:18.755357 7936 heartbeater.cc:347] Connected to a master server at 127.3.12.126:32809
I20260805 23:05:18.755625 7936 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:18.756153 7936 heartbeater.cc:511] Master 127.3.12.126:32809 requested a full tablet report, sending...
I20260805 23:05:18.757869 7637 ts_manager.cc:194] Registered new tserver with Master: 8d9839cd07b14087b1dfb49def5ae2b5 (127.3.12.65:37615)
I20260805 23:05:18.759645 7637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:43871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:18.772658 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:32809
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:18.885680 7943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:18.885957 7943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:18.886001 7943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:18.889987 7943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:18.890044 7943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:18.890103 7943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:18.895356 7943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:32809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:18.896461 7943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:18.897452 7943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:05:18.907401 7943 server_base.cc:1034] running on GCE node
W20260805 23:05:18.907330 7957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:18.907337 7953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:18.907341 7954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:18.908200 7943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:18.908919 7943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:18.910211 7943 hybrid_clock.cc:648] HybridClock initialized: now 1785971118910195 us; error 32 us; skew 500 ppm
Aug 05 23:05:18 dist-test-slave-2d19 krb5kdc[7569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:18.913411 7943 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:18.914377 7943 webserver.cc:460] Webserver started at http://127.3.12.66:36699/ using document root <none> and password file <none>
I20260805 23:05:18.914922 7943 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:18.914981 7943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:18.915148 7943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:18.917323 7943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "6544298b26ae484584885a108d74dc12"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "315ee8dee5780cad4f6e761bd0c4cf31"
server_key_iv: "388dbbf3ba795b9b870c861051884264"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.917853 7943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "6544298b26ae484584885a108d74dc12"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "315ee8dee5780cad4f6e761bd0c4cf31"
server_key_iv: "388dbbf3ba795b9b870c861051884264"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.921586 7943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:18.923965 7968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:18.925272 7943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:18.925369 7943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "6544298b26ae484584885a108d74dc12"
format_stamp: "Formatted at 2026-08-05 23:05:18 on dist-test-slave-2d19"
server_key: "315ee8dee5780cad4f6e761bd0c4cf31"
server_key_iv: "388dbbf3ba795b9b870c861051884264"
server_key_version: "encryptionkey@0"
I20260805 23:05:18.925451 7943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:18.981863 7943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:18.985340 7943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:18.985507 7943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:18.986084 7943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:18.986912 7943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:18.986961 7943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:18.987004 7943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:18.987020 7943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:18.996299 7943 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:45151
I20260805 23:05:18.996345 8125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:45151 every 8 connection(s)
I20260805 23:05:18.997254 7943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:19.000833 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 7943
I20260805 23:05:19.000926 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:19.001204 3121 external_mini_cluster.cc:1428] Setting key 1b74c2f4cf52268765445c31faeee51b
Aug 05 23:05:19 dist-test-slave-2d19 krb5kdc[7569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971118, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:05:19.012810 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:18.999236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:55513 (local address 127.3.12.126:32809)
0805 23:05:18.999386 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:18.999390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:19.000237 (+ 847us) server_negotiation.cc:407] Connection header received
0805 23:05:19.001845 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:19.001848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:19.001917 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:19.002003 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:19.003324 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.003852 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.005495 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.005724 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.005811 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:19.008863 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:19.008867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:19.008869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:19.008895 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:19.010454 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:19.011018 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:19.011021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:19.011022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:19.011102 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:19.011470 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:19.011473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:19.011475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:19.011668 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:19.011775 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:19.012471 (+ 696us) server_negotiation.cc:299] Negotiation successful
0805 23:05:19.012628 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:05:19.013785 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:18.999591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:32809 (local address 127.3.12.66:55513)
0805 23:05:19.000074 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:19.000120 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:19.001600 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:19.002172 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:19.002179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:19.002662 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:19.003173 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:19.003188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.003998 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.004002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:19.005348 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:19.005361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.005888 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.005895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:19.006132 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:19.006786 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:19.006813 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:19.008677 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:19.010588 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:19.010595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:19.010612 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:19.010899 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:19.011207 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:19.011211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:19.011213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:19.011367 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:19.011777 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:19.011786 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:19.012108 (+ 322us) client_negotiation.cc:769] Sending connection context
0805 23:05:19.012390 (+ 282us) client_negotiation.cc:240] Negotiation successful
0805 23:05:19.012787 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
I20260805 23:05:19.015112 8126 heartbeater.cc:347] Connected to a master server at 127.3.12.126:32809
I20260805 23:05:19.015374 8126 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:19.015954 8126 heartbeater.cc:511] Master 127.3.12.126:32809 requested a full tablet report, sending...
I20260805 23:05:19.017253 7637 ts_manager.cc:194] Registered new tserver with Master: 6544298b26ae484584885a108d74dc12 (127.3.12.66:45151)
I20260805 23:05:19.018187 7637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:55513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:19.031734 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:32809
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:19.145138 8135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:19.145401 8135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:19.145450 8135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:19.151629 8135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:19.151715 8135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:19.151810 8135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:19.157919 8135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:32809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:19.159482 8135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:19.160693 8135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:19.171619 8142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:19.171809 8135 server_base.cc:1034] running on GCE node
W20260805 23:05:19.171864 8146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:19.171619 8143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:19.172587 8135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:19.173358 8135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:19.174594 8135 hybrid_clock.cc:648] HybridClock initialized: now 1785971119174555 us; error 58 us; skew 500 ppm
Aug 05 23:05:19 dist-test-slave-2d19 krb5kdc[7569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971119, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:19.178468 8135 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:19.179718 8135 webserver.cc:460] Webserver started at http://127.3.12.67:32821/ using document root <none> and password file <none>
I20260805 23:05:19.180398 8135 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:19.180505 8135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:19.180728 8135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:19.182891 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "26aa62836e714bde809afb09759c1632"
format_stamp: "Formatted at 2026-08-05 23:05:19 on dist-test-slave-2d19"
server_key: "edf2384bd6118d6fb5661d19091bc174"
server_key_iv: "5febec6c8dc58290a3224775fbf47b4d"
server_key_version: "encryptionkey@0"
I20260805 23:05:19.183486 8135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "26aa62836e714bde809afb09759c1632"
format_stamp: "Formatted at 2026-08-05 23:05:19 on dist-test-slave-2d19"
server_key: "edf2384bd6118d6fb5661d19091bc174"
server_key_iv: "5febec6c8dc58290a3224775fbf47b4d"
server_key_version: "encryptionkey@0"
I20260805 23:05:19.187554 8135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:05:19.190392 8156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:19.191893 8135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260805 23:05:19.192070 8135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "26aa62836e714bde809afb09759c1632"
format_stamp: "Formatted at 2026-08-05 23:05:19 on dist-test-slave-2d19"
server_key: "edf2384bd6118d6fb5661d19091bc174"
server_key_iv: "5febec6c8dc58290a3224775fbf47b4d"
server_key_version: "encryptionkey@0"
I20260805 23:05:19.192183 8135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:19.206043 8135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:19.209347 8135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:19.209558 8135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:19.210160 8135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:19.211018 8135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:19.211088 8135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:19.211158 8135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:19.211203 8135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:19.221988 8135 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:43849
I20260805 23:05:19.222046 8324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:43849 every 8 connection(s)
I20260805 23:05:19.222936 8135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:19.228801 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 8135
I20260805 23:05:19.228945 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:19.229296 3121 external_mini_cluster.cc:1428] Setting key c7d81261fc3ba7459f4c37332331eb5e
Aug 05 23:05:19 dist-test-slave-2d19 krb5kdc[7569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971119, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:19.241724 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:19.225085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:33489 (local address 127.3.12.126:32809)
0805 23:05:19.225253 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:19.225258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:19.226298 (+ 1040us) server_negotiation.cc:407] Connection header received
0805 23:05:19.228353 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:19.228356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:19.228422 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:19.228549 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:19.230287 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.231154 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.233129 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.233555 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.233690 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:19.237111 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:19.237115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:19.237118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:19.237152 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:19.239083 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:19.239743 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:19.239749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:19.239752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:19.239853 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:19.240214 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:19.240218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:19.240220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:19.240463 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:19.240584 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:19.241331 (+ 747us) server_negotiation.cc:299] Negotiation successful
0805 23:05:19.241489 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 23:05:19.242574 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:19.225561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:32809 (local address 127.3.12.67:33489)
0805 23:05:19.226099 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:19.226162 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:19.228071 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:19.228754 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:19.228764 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:19.229293 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:19.230053 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:19.230130 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.231359 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.231363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:19.232988 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:19.233002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.233747 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.233753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:19.234010 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:19.234792 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:19.234814 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:19.236904 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:19.239208 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:19.239215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:19.239231 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:19.239610 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:19.239967 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:19.239969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:19.239970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:19.240102 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:19.240646 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:19.240653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:19.241006 (+ 353us) client_negotiation.cc:769] Sending connection context
0805 23:05:19.241211 (+ 205us) client_negotiation.cc:240] Negotiation successful
0805 23:05:19.241504 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":129,"threads_started":1}
I20260805 23:05:19.244168 8325 heartbeater.cc:347] Connected to a master server at 127.3.12.126:32809
I20260805 23:05:19.244459 8325 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:19.244972 8325 heartbeater.cc:511] Master 127.3.12.126:32809 requested a full tablet report, sending...
I20260805 23:05:19.246043 7637 ts_manager.cc:194] Registered new tserver with Master: 26aa62836e714bde809afb09759c1632 (127.3.12.67:43849)
I20260805 23:05:19.246654 7637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:33489
I20260805 23:05:19.256618 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 23:05:19 dist-test-slave-2d19 krb5kdc[7569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971119, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 05 23:05:19 dist-test-slave-2d19 krb5kdc[7569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971119, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:19.285784 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:19.276801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.3.12.126:32809)
0805 23:05:19.276957 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:19.276963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:19.277139 (+ 176us) server_negotiation.cc:407] Connection header received
0805 23:05:19.277315 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:19.277317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:19.277363 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:19.277447 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:19.278174 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.278704 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.279549 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:19.279769 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:19.279872 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:19.282678 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:19.282681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:19.282683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:19.282710 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:19.283994 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:19.284482 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:19.284486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:19.284488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:19.284569 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:19.284843 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:19.284847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:19.284849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:19.285110 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:19.285257 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:19.285489 (+ 232us) server_negotiation.cc:299] Negotiation successful
0805 23:05:19.285608 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260805 23:05:19.286180 7637 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33204
I20260805 23:05:19.287029 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 7733
I20260805 23:05:19.303393 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 7943
I20260805 23:05:19.319743 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 8135
I20260805 23:05:19.330288 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 7594
2026-08-05T23:05:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2448 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:19 dist-test-slave-2d19 krb5kdc[8351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:19 dist-test-slave-2d19 krb5kdc[8351](info): set up 2 sockets
Aug 05 23:05:19 dist-test-slave-2d19 krb5kdc[8351](info): commencing operation
krb5kdc: starting...
W20260805 23:05:20.560611 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:20 dist-test-slave-2d19 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:20.695731 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41795
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:20.811496 8372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:20.811909 8372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:20.812040 8372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:20.816677 8372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:20.816777 8372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:20.816818 8372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:20.816871 8372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:20.816906 8372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:20.822691 8372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:20.823899 8372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:20.825002 8372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:20.833927 8378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:20.833935 8380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:20.833846 8382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:20.834239 8372 server_base.cc:1034] running on GCE node
I20260805 23:05:20.834874 8372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:20.836048 8372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:20.837270 8372 hybrid_clock.cc:648] HybridClock initialized: now 1785971120837262 us; error 53 us; skew 500 ppm
Aug 05 23:05:20 dist-test-slave-2d19 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971120, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:20.840101 8372 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:20.841082 8372 webserver.cc:460] Webserver started at http://127.3.12.126:35003/ using document root <none> and password file <none>
I20260805 23:05:20.841612 8372 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:20.841675 8372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:20.841845 8372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:20.843878 8372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7"
format_stamp: "Formatted at 2026-08-05 23:05:20 on dist-test-slave-2d19"
server_key: "89f7f1bef9a3e1735aad714e76137084"
server_key_iv: "8d18fc754341aa4c432be63c4901da38"
server_key_version: "encryptionkey@0"
I20260805 23:05:20.844475 8372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7"
format_stamp: "Formatted at 2026-08-05 23:05:20 on dist-test-slave-2d19"
server_key: "89f7f1bef9a3e1735aad714e76137084"
server_key_iv: "8d18fc754341aa4c432be63c4901da38"
server_key_version: "encryptionkey@0"
I20260805 23:05:20.848148 8372 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:05:20.850631 8390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:20.851965 8372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:20.852134 8372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7"
format_stamp: "Formatted at 2026-08-05 23:05:20 on dist-test-slave-2d19"
server_key: "89f7f1bef9a3e1735aad714e76137084"
server_key_iv: "8d18fc754341aa4c432be63c4901da38"
server_key_version: "encryptionkey@0"
I20260805 23:05:20.852238 8372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:20.861848 8372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:20.864701 8372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:20.864897 8372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:20.872041 8372 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:41795
I20260805 23:05:20.872059 8466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:41795 every 8 connection(s)
I20260805 23:05:20.873100 8372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:20.875849 8467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:20.881376 8467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7: Bootstrap starting.
I20260805 23:05:20.883042 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 8372
I20260805 23:05:20.883188 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:20.883504 3121 external_mini_cluster.cc:1428] Setting key a3dddb94d389cb5970875b645c395aae
I20260805 23:05:20.884142 8467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:20.885113 8467 log.cc:824] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:20.887274 8467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7: No bootstrap required, opened a new log
I20260805 23:05:20.891815 8467 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41795 } }
I20260805 23:05:20.892252 8467 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:20.892374 8467 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4cfa1d744f40dea19d3ed0d901b7a7, State: Initialized, Role: FOLLOWER
Aug 05 23:05:20 dist-test-slave-2d19 krb5kdc[8351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:20.892935 8467 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41795 } }
I20260805 23:05:20.893131 8467 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:20.893218 8467 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:20.893337 8467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:20.894379 8467 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41795 } }
I20260805 23:05:20.894755 8467 leader_election.cc:302] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea4cfa1d744f40dea19d3ed0d901b7a7; no voters:
I20260805 23:05:20.895179 8467 leader_election.cc:288] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:20.895267 8473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:20.895475 8473 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [term 1 LEADER]: Becoming Leader. State: Replica: ea4cfa1d744f40dea19d3ed0d901b7a7, State: Running, Role: LEADER
I20260805 23:05:20.895792 8473 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41795 } }
I20260805 23:05:20.896675 8467 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:20.897529 8476 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4cfa1d744f40dea19d3ed0d901b7a7. Latest consensus state: current_term: 1 leader_uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41795 } } }
I20260805 23:05:20.897675 8476 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:20.898124 8475 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4cfa1d744f40dea19d3ed0d901b7a7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41795 } } }
I20260805 23:05:20.898456 8475 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:20.899891 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:20.884722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.3.12.126:41795)
0805 23:05:20.885143 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:20.885159 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:20.885217 (+ 58us) server_negotiation.cc:407] Connection header received
0805 23:05:20.886036 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:20.886055 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:20.886387 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:20.886678 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:20.887611 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:20.888655 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:20.889846 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:20.890337 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:20.890458 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:20.893667 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:20.893672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:20.893688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:20.893720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:20.895817 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:20.896409 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:20.896414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:20.896419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:20.896535 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:20.896868 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:20.896872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:20.896878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:20.897513 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:20.898417 (+ 904us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:20.898606 (+ 189us) server_negotiation.cc:299] Negotiation successful
0805 23:05:20.898929 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":122,"threads_started":1}
W20260805 23:05:20.902341 8494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:05:20.902432 8494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:05:20.902494 8488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:20.903342 8488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:20.909147 8488 catalog_manager.cc:1347] Generated new cluster ID: 9987bd300d9f4a37adc7b14d42840b18
I20260805 23:05:20.909217 8488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:20.925424 8488 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:20.926482 8488 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:20.935719 8488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea4cfa1d744f40dea19d3ed0d901b7a7: Generated new TSK 0
I20260805 23:05:20.936823 8488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:05:20.979238 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41795
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:21.102046 8503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:21.102312 8503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:21.102357 8503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:21.106343 8503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:21.106403 8503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:21.106493 8503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:21.112166 8503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:21.113278 8503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:21.114378 8503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:21.124310 8512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:21.124316 8510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:21.124385 8514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:21.124598 8503 server_base.cc:1034] running on GCE node
I20260805 23:05:21.125099 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:21.125716 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:21.126914 8503 hybrid_clock.cc:648] HybridClock initialized: now 1785971121126896 us; error 27 us; skew 500 ppm
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:21.129752 8503 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:05:21.130842 8503 webserver.cc:460] Webserver started at http://127.3.12.65:46587/ using document root <none> and password file <none>
I20260805 23:05:21.131506 8503 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:21.131603 8503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:21.131848 8503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:21.134068 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "04bb05c77f84455eb4f224643e3a213c"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "84886bfa440fd47c2bdff3d0a58b5e7d"
server_key_iv: "567d9784e56dcfc71b15ca27702fc461"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.134660 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "04bb05c77f84455eb4f224643e3a213c"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "84886bfa440fd47c2bdff3d0a58b5e7d"
server_key_iv: "567d9784e56dcfc71b15ca27702fc461"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.138535 8503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260805 23:05:21.140986 8524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.142130 8503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:05:21.142262 8503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "04bb05c77f84455eb4f224643e3a213c"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "84886bfa440fd47c2bdff3d0a58b5e7d"
server_key_iv: "567d9784e56dcfc71b15ca27702fc461"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.142370 8503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:21.158015 8503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:21.160732 8503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:21.160936 8503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:21.161506 8503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:21.162335 8503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:21.162407 8503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.162480 8503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:21.162521 8503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.171985 8503 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:36033
I20260805 23:05:21.172008 8692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:36033 every 8 connection(s)
I20260805 23:05:21.172953 8503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:21.177745 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 8503
I20260805 23:05:21.177891 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:21.178206 3121 external_mini_cluster.cc:1428] Setting key aea241d06e25fe5601f5d9fa8fa17457
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:21.188638 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:21.174819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:35369 (local address 127.3.12.126:41795)
0805 23:05:21.174962 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:21.174966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:21.175864 (+ 898us) server_negotiation.cc:407] Connection header received
0805 23:05:21.177337 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:21.177341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:21.177401 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:21.177509 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:21.178944 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.179530 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.180792 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.181213 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.181343 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:21.184598 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:21.184604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:21.184607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:21.184635 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:21.186433 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.186987 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.186991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.186993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.187077 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.187440 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.187443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.187445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.187640 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:21.187745 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:21.188301 (+ 556us) server_negotiation.cc:299] Negotiation successful
0805 23:05:21.188455 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260805 23:05:21.189471 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:21.175272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41795 (local address 127.3.12.65:35369)
0805 23:05:21.175718 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:21.175761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:21.177123 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:21.177708 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:21.177716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:21.178161 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:21.178800 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:21.178819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.179668 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.179672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:21.180668 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:21.180679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.181371 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.181374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:21.181557 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:21.182306 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:21.182340 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:21.184351 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:21.186564 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:21.186571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:21.186588 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:21.186866 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:21.187173 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:21.187176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:21.187178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:21.187329 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:21.187746 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:21.187755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:21.188028 (+ 273us) client_negotiation.cc:769] Sending connection context
0805 23:05:21.188215 (+ 187us) client_negotiation.cc:240] Negotiation successful
0805 23:05:21.188592 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:21.190737 8694 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41795
I20260805 23:05:21.190994 8694 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:21.191493 8694 heartbeater.cc:511] Master 127.3.12.126:41795 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:05:21.193266 8412 ts_manager.cc:194] Registered new tserver with Master: 04bb05c77f84455eb4f224643e3a213c (127.3.12.65:36033)
I20260805 23:05:21.194389 8412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:35369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:21.211485 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41795
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:21.327255 8702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:21.327497 8702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:21.327540 8702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:21.331594 8702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:21.331660 8702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:21.331717 8702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:21.337015 8702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:21.338124 8702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:21.339124 8702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:21.349068 8710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:21.349076 8713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:21.349010 8711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:21.349398 8702 server_base.cc:1034] running on GCE node
I20260805 23:05:21.349869 8702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:21.350605 8702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:21.351804 8702 hybrid_clock.cc:648] HybridClock initialized: now 1785971121351784 us; error 36 us; skew 500 ppm
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:21.354772 8702 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:21.355669 8702 webserver.cc:460] Webserver started at http://127.3.12.66:46179/ using document root <none> and password file <none>
I20260805 23:05:21.356191 8702 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:21.356344 8702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:21.356592 8702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:21.358644 8702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "a2ca038757a342a39bde5be381df44c2"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "2545a7dc7f417e0a8459583f8feb6941"
server_key_iv: "c60664495a993394c93ea065ee845366"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.359220 8702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "a2ca038757a342a39bde5be381df44c2"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "2545a7dc7f417e0a8459583f8feb6941"
server_key_iv: "c60664495a993394c93ea065ee845366"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.363073 8702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260805 23:05:21.365584 8727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.366772 8702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:21.366895 8702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "a2ca038757a342a39bde5be381df44c2"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "2545a7dc7f417e0a8459583f8feb6941"
server_key_iv: "c60664495a993394c93ea065ee845366"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.367023 8702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:21.377761 8702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:21.381072 8702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:21.381227 8702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:21.381759 8702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:21.382534 8702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:21.382581 8702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.382622 8702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:21.382643 8702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.391616 8702 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:33179
I20260805 23:05:21.391737 8894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:33179 every 8 connection(s)
I20260805 23:05:21.392665 8702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:21.398913 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 8702
I20260805 23:05:21.399062 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:21.399353 3121 external_mini_cluster.cc:1428] Setting key 0f6f8df6556b5420ae737215a5c1436b
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:21.407850 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:21.394540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:37299 (local address 127.3.12.126:41795)
0805 23:05:21.394678 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:21.394682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:21.395551 (+ 869us) server_negotiation.cc:407] Connection header received
0805 23:05:21.397059 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:21.397063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:21.397120 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:21.397225 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:21.398484 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.399141 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.400477 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.400776 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.400903 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:21.403591 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:21.403598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:21.403603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:21.403642 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:21.405381 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.406123 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.406128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.406132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.406232 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.406671 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.406674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.406675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.406858 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:21.406944 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:21.407621 (+ 677us) server_negotiation.cc:299] Negotiation successful
0805 23:05:21.407728 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260805 23:05:21.409152 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:21.394949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41795 (local address 127.3.12.66:37299)
0805 23:05:21.395397 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:21.395440 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:21.396791 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:21.397431 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:21.397438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:21.397787 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:21.398336 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:21.398351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.399267 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.399276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:21.400358 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:21.400369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.400942 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.400947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:21.401110 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:21.401756 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:21.401772 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:21.403412 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:21.405523 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:21.405532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:21.405549 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:21.405985 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:21.406367 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:21.406372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:21.406376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:21.406561 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:21.406981 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:21.406991 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:21.407344 (+ 353us) client_negotiation.cc:769] Sending connection context
0805 23:05:21.407594 (+ 250us) client_negotiation.cc:240] Negotiation successful
0805 23:05:21.407997 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:05:21.410529 8895 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41795
I20260805 23:05:21.410802 8895 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:21.411319 8895 heartbeater.cc:511] Master 127.3.12.126:41795 requested a full tablet report, sending...
I20260805 23:05:21.412441 8412 ts_manager.cc:194] Registered new tserver with Master: a2ca038757a342a39bde5be381df44c2 (127.3.12.66:33179)
I20260805 23:05:21.413036 8412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:37299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:21.431253 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41795
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:44863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:21.547207 8901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:21.547482 8901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:21.547564 8901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:21.551757 8901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:21.551826 8901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:21.551935 8901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:21.557551 8901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:44863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:41795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:21.558702 8901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:21.559782 8901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:21.569766 8913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:21.569766 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:21.569809 8915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:21.570012 8901 server_base.cc:1034] running on GCE node
I20260805 23:05:21.570722 8901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:21.571406 8901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:21.572611 8901 hybrid_clock.cc:648] HybridClock initialized: now 1785971121572577 us; error 51 us; skew 500 ppm
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:21.579545 8901 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:21.580605 8901 webserver.cc:460] Webserver started at http://127.3.12.67:38249/ using document root <none> and password file <none>
I20260805 23:05:21.581158 8901 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:21.581218 8901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:21.581385 8901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:21.583432 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "820e9af745dd48df831a10aff49c4407"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "45aaeca1724bbef39024de4cdf5d8a56"
server_key_iv: "2533b69d338d26bd671b315316f0832d"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.583973 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "820e9af745dd48df831a10aff49c4407"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "45aaeca1724bbef39024de4cdf5d8a56"
server_key_iv: "2533b69d338d26bd671b315316f0832d"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.587874 8901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:05:21.590358 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.591769 8901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260805 23:05:21.591907 8901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "820e9af745dd48df831a10aff49c4407"
format_stamp: "Formatted at 2026-08-05 23:05:21 on dist-test-slave-2d19"
server_key: "45aaeca1724bbef39024de4cdf5d8a56"
server_key_iv: "2533b69d338d26bd671b315316f0832d"
server_key_version: "encryptionkey@0"
I20260805 23:05:21.592020 8901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:21.610129 8901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:21.613313 8901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:21.613521 8901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:21.614128 8901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:21.614975 8901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:21.615022 8901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.615087 8901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:21.615121 8901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:21.624852 8901 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:37913
I20260805 23:05:21.624888 9091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:37913 every 8 connection(s)
I20260805 23:05:21.625744 8901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:21.628710 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 8901
I20260805 23:05:21.628837 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:21.629158 3121 external_mini_cluster.cc:1428] Setting key 6f80c68b586194d9ba0ef466f577a07c
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:21.639585 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:21.627546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:36325 (local address 127.3.12.126:41795)
0805 23:05:21.627694 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:21.627698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:21.628582 (+ 884us) server_negotiation.cc:407] Connection header received
0805 23:05:21.630262 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:21.630266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:21.630322 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:21.630425 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:21.631581 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.632131 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.633300 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.633530 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.633651 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:21.636109 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:21.636112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:21.636114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:21.636139 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:21.637534 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.638053 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.638057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.638059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.638150 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.638499 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.638503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.638505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.638693 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:21.638804 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:21.639292 (+ 488us) server_negotiation.cc:299] Negotiation successful
0805 23:05:21.639417 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 23:05:21.640499 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:21.627924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41795 (local address 127.3.12.67:36325)
0805 23:05:21.628431 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:21.628475 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:21.630041 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:21.630604 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:21.630612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:21.630967 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:21.631431 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:21.631444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.632285 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.632288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:21.633167 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:21.633176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.633691 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.633695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:21.633864 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:21.634467 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:21.634483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:21.635928 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:21.637657 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:21.637664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:21.637680 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:21.637945 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:21.638256 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:21.638259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:21.638262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:21.638401 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:21.638801 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:21.638808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:21.639055 (+ 247us) client_negotiation.cc:769] Sending connection context
0805 23:05:21.639250 (+ 195us) client_negotiation.cc:240] Negotiation successful
0805 23:05:21.639596 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260805 23:05:21.641662 9092 heartbeater.cc:347] Connected to a master server at 127.3.12.126:41795
I20260805 23:05:21.641938 9092 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:21.642458 9092 heartbeater.cc:511] Master 127.3.12.126:41795 requested a full tablet report, sending...
I20260805 23:05:21.643539 8412 ts_manager.cc:194] Registered new tserver with Master: 820e9af745dd48df831a10aff49c4407 (127.3.12.67:37913)
I20260805 23:05:21.644112 8412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:36325
I20260805 23:05:21.654845 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:05:21.683135 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:21.673537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.3.12.65:36033)
0805 23:05:21.673850 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:21.673865 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:21.673892 (+ 27us) server_negotiation.cc:407] Connection header received
0805 23:05:21.673945 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:21.673950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:21.674152 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:21.674284 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:21.674966 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.675788 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.676726 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.677140 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.677228 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:21.679752 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:21.679762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:21.679777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:21.679800 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:21.681507 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.681991 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.681996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.681997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.682075 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.682297 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.682299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.682300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.682538 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:21.682726 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:21.682849 (+ 123us) server_negotiation.cc:299] Negotiation successful
0805 23:05:21.682982 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[8351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:21.694875 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:21.686171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.3.12.126:41795)
0805 23:05:21.686320 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:21.686324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:21.686429 (+ 105us) server_negotiation.cc:407] Connection header received
0805 23:05:21.686586 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:21.686590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:21.686638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:21.686722 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:21.687418 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.688379 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.689124 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:21.689337 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:21.689423 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:21.691755 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:21.691758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:21.691760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:21.691785 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:21.693323 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.693711 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.693714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.693716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.693797 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:21.694049 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:21.694052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:21.694054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:21.694275 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:21.694392 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:21.694562 (+ 170us) server_negotiation.cc:299] Negotiation successful
0805 23:05:21.694656 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260805 23:05:21.695333 8412 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33146
I20260805 23:05:21.696085 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 8503
I20260805 23:05:21.718374 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 8702
I20260805 23:05:21.729030 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 8901
I20260805 23:05:21.740839 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 8372
2026-08-05T23:05:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2405 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[9109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[9109](info): set up 2 sockets
Aug 05 23:05:21 dist-test-slave-2d19 krb5kdc[9109](info): commencing operation
krb5kdc: starting...
W20260805 23:05:22.971452 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:23 dist-test-slave-2d19 krb5kdc[9109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:23.107947 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:39893
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:39893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:23.225148 9129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:23.225467 9129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:23.225551 9129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:23.229660 9129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:23.229758 9129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:23.229799 9129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:23.229833 9129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:23.229871 9129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:23.237867 9129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:39893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:39893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:23.239076 9129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:23.240155 9129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:23.248898 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:23.248903 9141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:23.248915 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:23.249344 9129 server_base.cc:1034] running on GCE node
I20260805 23:05:23.249922 9129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:23.251106 9129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:23.252328 9129 hybrid_clock.cc:648] HybridClock initialized: now 1785971123252311 us; error 31 us; skew 500 ppm
Aug 05 23:05:23 dist-test-slave-2d19 krb5kdc[9109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971123, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:23.255184 9129 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:23.255985 9129 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:23.256095 9129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:23.256377 9129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:23.258457 9129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "fcfe234c007a4b3f95fa6e48ee7d261a"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "c3afb94f5ac2562efb2ecd640fe6a0c0"
server_key_iv: "b7797c2278c243fae93dd6fd2c75fb47"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.259042 9129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "fcfe234c007a4b3f95fa6e48ee7d261a"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "c3afb94f5ac2562efb2ecd640fe6a0c0"
server_key_iv: "b7797c2278c243fae93dd6fd2c75fb47"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.262858 9129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:05:23.265373 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:23.266607 9129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:23.266774 9129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "fcfe234c007a4b3f95fa6e48ee7d261a"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "c3afb94f5ac2562efb2ecd640fe6a0c0"
server_key_iv: "b7797c2278c243fae93dd6fd2c75fb47"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.266891 9129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:23.283193 9129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:23.286370 9129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:23.286582 9129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:23.293819 9129 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:39893
I20260805 23:05:23.294103 9221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:39893 every 8 connection(s)
I20260805 23:05:23.294523 9129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:23.295409 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 9129
I20260805 23:05:23.295578 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:23.295878 3121 external_mini_cluster.cc:1428] Setting key e985936570e87c04d104e74e25cc8aea
I20260805 23:05:23.297876 9223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:23.304713 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a: Bootstrap starting.
Aug 05 23:05:23 dist-test-slave-2d19 krb5kdc[9109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:23.307530 9223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:23.308501 9223 log.cc:824] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:23.310518 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a: No bootstrap required, opened a new log
I20260805 23:05:23.311182 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:23.297516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.3.12.126:39893)
0805 23:05:23.298072 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:23.298089 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:23.298127 (+ 38us) server_negotiation.cc:407] Connection header received
0805 23:05:23.298798 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:23.298814 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:23.299153 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:23.299439 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:23.300419 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.301407 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.302569 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.303024 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.303208 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:23.306118 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:23.306125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:23.306142 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:23.306174 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:23.308220 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:23.308694 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:23.308699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:23.308703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:23.308819 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:23.309133 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:23.309138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:23.309144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:23.309532 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:23.309766 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:23.310066 (+ 300us) server_negotiation.cc:299] Negotiation successful
0805 23:05:23.310348 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20260805 23:05:23.315014 9223 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe234c007a4b3f95fa6e48ee7d261a" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39893 } }
I20260805 23:05:23.315258 9223 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:23.315352 9223 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfe234c007a4b3f95fa6e48ee7d261a, State: Initialized, Role: FOLLOWER
I20260805 23:05:23.315800 9223 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe234c007a4b3f95fa6e48ee7d261a" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39893 } }
I20260805 23:05:23.315973 9223 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:23.316061 9223 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:23.316160 9223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:23.317205 9223 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe234c007a4b3f95fa6e48ee7d261a" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39893 } }
I20260805 23:05:23.317564 9223 leader_election.cc:302] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcfe234c007a4b3f95fa6e48ee7d261a; no voters:
I20260805 23:05:23.317934 9223 leader_election.cc:288] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:23.318055 9233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:23.318315 9233 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [term 1 LEADER]: Becoming Leader. State: Replica: fcfe234c007a4b3f95fa6e48ee7d261a, State: Running, Role: LEADER
I20260805 23:05:23.318612 9233 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe234c007a4b3f95fa6e48ee7d261a" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39893 } }
I20260805 23:05:23.319185 9223 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:23.320225 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcfe234c007a4b3f95fa6e48ee7d261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe234c007a4b3f95fa6e48ee7d261a" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39893 } } }
I20260805 23:05:23.320391 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:23.320808 9244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:23.320984 9236 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcfe234c007a4b3f95fa6e48ee7d261a. Latest consensus state: current_term: 1 leader_uuid: "fcfe234c007a4b3f95fa6e48ee7d261a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfe234c007a4b3f95fa6e48ee7d261a" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39893 } } }
I20260805 23:05:23.321098 9236 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:23.323693 9244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:23.329699 9244 catalog_manager.cc:1347] Generated new cluster ID: f07b852f9f704834b9fab64c891a1922
I20260805 23:05:23.329774 9244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:23.365366 9244 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:23.366515 9244 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:23.373312 9244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a: Generated new TSK 0
I20260805 23:05:23.374058 9244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:05:23.413547 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39893
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:23.530485 9256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:23.530829 9256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:23.530912 9256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:23.535037 9256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:23.535149 9256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:23.535251 9256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:23.540783 9256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:23.541932 9256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:23.543028 9256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:23.552780 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:23.552786 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:23.552799 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:23.553119 9256 server_base.cc:1034] running on GCE node
I20260805 23:05:23.553553 9256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:23.554229 9256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:23.555418 9256 hybrid_clock.cc:648] HybridClock initialized: now 1785971123555397 us; error 34 us; skew 500 ppm
Aug 05 23:05:23 dist-test-slave-2d19 krb5kdc[9109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971123, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:23.558429 9256 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:05:23.559201 9256 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:23.559269 9256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:23.559441 9256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:23.561527 9256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "93ca203f7270444688c9b574a4c903d1"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "fec5ff0bb65a83214c7de8a7c760391c"
server_key_iv: "d63b58230aac392209ef6bd6a6a5d4f6"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.562052 9256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "93ca203f7270444688c9b574a4c903d1"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "fec5ff0bb65a83214c7de8a7c760391c"
server_key_iv: "d63b58230aac392209ef6bd6a6a5d4f6"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.565831 9256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260805 23:05:23.568418 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:23.569993 9256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:05:23.570111 9256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "93ca203f7270444688c9b574a4c903d1"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "fec5ff0bb65a83214c7de8a7c760391c"
server_key_iv: "d63b58230aac392209ef6bd6a6a5d4f6"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.570235 9256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:23.582087 9256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:23.585433 9256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:23.585588 9256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:23.586131 9256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:23.586932 9256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:23.586980 9256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:23.587046 9256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:23.587070 9256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:23.596648 9256 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:43757
I20260805 23:05:23.596688 9437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:43757 every 8 connection(s)
I20260805 23:05:23.597289 9256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:23.601265 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 9256
I20260805 23:05:23.601369 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:23.601699 3121 external_mini_cluster.cc:1428] Setting key d4efd5219c70a90b6657c28ded4a1336
Aug 05 23:05:23 dist-test-slave-2d19 krb5kdc[9109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971123, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:23.612864 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:23.599114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:43899 (local address 127.3.12.126:39893)
0805 23:05:23.599296 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:23.599301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:23.600203 (+ 902us) server_negotiation.cc:407] Connection header received
0805 23:05:23.601642 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:23.601646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:23.601707 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:23.601815 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:23.603181 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.603756 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.605580 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.605989 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.606154 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:23.608696 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:23.608701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:23.608705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:23.608737 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:23.610499 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:23.611041 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:23.611048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:23.611050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:23.611156 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:23.611509 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:23.611515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:23.611518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:23.611784 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:23.611915 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:23.612504 (+ 589us) server_negotiation.cc:299] Negotiation successful
0805 23:05:23.612668 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:23.613634 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:23.599549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:39893 (local address 127.3.12.65:43899)
0805 23:05:23.600034 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:23.600087 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:23.601426 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:23.602000 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:23.602007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:23.602377 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:23.603037 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:23.603053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.603906 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.603912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:23.605427 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:23.605444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.606178 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.606180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:23.606329 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:23.606966 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:23.606982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:23.608507 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:23.610628 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:23.610634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:23.610647 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:23.610909 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:23.611253 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:23.611256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:23.611257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:23.611392 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:23.611895 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:23.611911 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:23.612172 (+ 261us) client_negotiation.cc:769] Sending connection context
0805 23:05:23.612389 (+ 217us) client_negotiation.cc:240] Negotiation successful
0805 23:05:23.612673 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260805 23:05:23.614992 9438 heartbeater.cc:347] Connected to a master server at 127.3.12.126:39893
I20260805 23:05:23.615258 9438 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:23.615770 9438 heartbeater.cc:511] Master 127.3.12.126:39893 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:05:23.617625 9175 ts_manager.cc:194] Registered new tserver with Master: 93ca203f7270444688c9b574a4c903d1 (127.3.12.65:43757)
I20260805 23:05:23.619022 9175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:43899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:23.635792 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39893
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:23.749354 9445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:23.749603 9445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:23.749650 9445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:23.753715 9445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:23.753778 9445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:23.753837 9445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:23.758996 9445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:23.759977 9445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:23.761010 9445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:23.770491 9455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:23.770522 9452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:23.770772 9453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:23.770915 9445 server_base.cc:1034] running on GCE node
I20260805 23:05:23.771378 9445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:23.772066 9445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:23.773273 9445 hybrid_clock.cc:648] HybridClock initialized: now 1785971123773238 us; error 75 us; skew 500 ppm
Aug 05 23:05:23 dist-test-slave-2d19 krb5kdc[9109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971123, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:23.776409 9445 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:23.777258 9445 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:23.777369 9445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:23.777638 9445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:23.779799 9445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "2dac0353aa4441969b68ea7da157ccd0"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "96bb0c1d48f1a0c3a02132b9166b35d0"
server_key_iv: "23f5105edb36d1f9dc6f3e15b1c136a2"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.780490 9445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "2dac0353aa4441969b68ea7da157ccd0"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "96bb0c1d48f1a0c3a02132b9166b35d0"
server_key_iv: "23f5105edb36d1f9dc6f3e15b1c136a2"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.784448 9445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:23.786953 9461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:23.788189 9445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:23.788429 9445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "2dac0353aa4441969b68ea7da157ccd0"
format_stamp: "Formatted at 2026-08-05 23:05:23 on dist-test-slave-2d19"
server_key: "96bb0c1d48f1a0c3a02132b9166b35d0"
server_key_iv: "23f5105edb36d1f9dc6f3e15b1c136a2"
server_key_version: "encryptionkey@0"
I20260805 23:05:23.788535 9445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:23.819634 9445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:23.822984 9445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:23.823194 9445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:23.823837 9445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:23.824790 9445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:23.824862 9445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:23.824936 9445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:23.824990 9445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:23.833784 9445 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:39043
I20260805 23:05:23.833813 9627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:39043 every 8 connection(s)
I20260805 23:05:23.834378 9445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:23.843758 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 9445
I20260805 23:05:23.843885 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
Aug 05 23:05:23 dist-test-slave-2d19 krb5kdc[9109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971123, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:23.844213 3121 external_mini_cluster.cc:1428] Setting key bc91263762db8ae98a0b18933c411ffa
I20260805 23:05:23.848856 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:23.836188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:59881 (local address 127.3.12.126:39893)
0805 23:05:23.836381 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:23.836385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:23.837224 (+ 839us) server_negotiation.cc:407] Connection header received
0805 23:05:23.838501 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:23.838503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:23.838549 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:23.838625 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:23.839827 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.840690 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.841927 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.842146 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.842232 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:23.844920 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:23.844923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:23.844925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:23.844954 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:23.846638 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:23.847134 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:23.847141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:23.847145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:23.847243 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:23.847566 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:23.847572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:23.847577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:23.847831 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:23.847953 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:23.848639 (+ 686us) server_negotiation.cc:299] Negotiation successful
0805 23:05:23.848741 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260805 23:05:23.850044 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:23.836599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:39893 (local address 127.3.12.66:59881)
0805 23:05:23.837056 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:23.837112 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:23.838302 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:23.838791 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:23.838797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:23.839164 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:23.839637 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:23.839650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.840841 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.840847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:23.841809 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:23.841820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:23.842281 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:23.842284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:23.842420 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:23.842980 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:23.842997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:23.844738 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:23.846758 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:23.846763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:23.846775 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:23.847031 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:23.847336 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:23.847338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:23.847340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:23.847473 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:23.847963 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:23.847973 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:23.848366 (+ 393us) client_negotiation.cc:769] Sending connection context
0805 23:05:23.848597 (+ 231us) client_negotiation.cc:240] Negotiation successful
0805 23:05:23.848975 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
I20260805 23:05:23.851482 9630 heartbeater.cc:347] Connected to a master server at 127.3.12.126:39893
I20260805 23:05:23.851768 9630 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:23.852361 9630 heartbeater.cc:511] Master 127.3.12.126:39893 requested a full tablet report, sending...
I20260805 23:05:23.853502 9175 ts_manager.cc:194] Registered new tserver with Master: 2dac0353aa4441969b68ea7da157ccd0 (127.3.12.66:39043)
I20260805 23:05:23.854071 9175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:59881
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:23.875371 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39893
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:23.988162 9639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:23.988497 9639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:23.988572 9639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:23.992667 9639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:23.992728 9639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:23.992837 9639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:23.998157 9639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:23.999248 9639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:24.000344 9639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:24.009994 9648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:24.010030 9644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:24.010193 9639 server_base.cc:1034] running on GCE node
W20260805 23:05:24.010030 9645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:24.010879 9639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:24.011564 9639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:24.012759 9639 hybrid_clock.cc:648] HybridClock initialized: now 1785971124012746 us; error 34 us; skew 500 ppm
Aug 05 23:05:24 dist-test-slave-2d19 krb5kdc[9109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971124, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:24.015705 9639 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:24.016501 9639 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:24.016571 9639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:24.016747 9639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:24.018934 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "38f462034dc74d1e8320655b75b832d7"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "113bf816ed64ef63f5327465c17f77c4"
server_key_iv: "bce82ebd0a68553d73033fbfb795dcb6"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.019482 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "38f462034dc74d1e8320655b75b832d7"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "113bf816ed64ef63f5327465c17f77c4"
server_key_iv: "bce82ebd0a68553d73033fbfb795dcb6"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.023655 9639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:05:24.026093 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.027323 9639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260805 23:05:24.027449 9639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "38f462034dc74d1e8320655b75b832d7"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "113bf816ed64ef63f5327465c17f77c4"
server_key_iv: "bce82ebd0a68553d73033fbfb795dcb6"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.027560 9639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:24.060441 9639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:24.063987 9639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:24.064217 9639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:24.064901 9639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:24.065784 9639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:24.065845 9639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.065887 9639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:24.065943 9639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.075860 9639 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:33347
I20260805 23:05:24.075919 9832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:33347 every 8 connection(s)
I20260805 23:05:24.076617 9639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:24.082638 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 9639
I20260805 23:05:24.082798 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:24.083209 3121 external_mini_cluster.cc:1428] Setting key 3b11d23cc74ec549df185e4feb555dee
Aug 05 23:05:24 dist-test-slave-2d19 krb5kdc[9109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971124, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:24.090786 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.078426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:44853 (local address 127.3.12.126:39893)
0805 23:05:24.078594 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.078599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.079497 (+ 898us) server_negotiation.cc:407] Connection header received
0805 23:05:24.080855 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.080858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.080904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.080982 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:24.082194 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.082719 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.084169 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.084466 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.084549 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.087095 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:24.087098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:24.087100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:24.087125 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:24.088712 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:24.089197 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:24.089199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:24.089200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:24.089284 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:24.089716 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:24.089718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:24.089720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:24.089894 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:24.089972 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.090545 (+ 573us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.090665 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 23:05:24.091688 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.078811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:39893 (local address 127.3.12.67:44853)
0805 23:05:24.079320 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.079365 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:24.080653 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:24.081188 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:24.081195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:24.081577 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:24.082062 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.082076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.082906 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.082910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.084047 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.084058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.084612 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.084619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.084768 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.085397 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:24.085420 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:24.086932 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:24.088821 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:24.088827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:24.088840 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:24.089110 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:24.089460 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:24.089463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:24.089465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:24.089622 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:24.090060 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:24.090067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:24.090323 (+ 256us) client_negotiation.cc:769] Sending connection context
0805 23:05:24.090501 (+ 178us) client_negotiation.cc:240] Negotiation successful
0805 23:05:24.090777 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260805 23:05:24.093021 9833 heartbeater.cc:347] Connected to a master server at 127.3.12.126:39893
I20260805 23:05:24.093325 9833 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:24.093796 9833 heartbeater.cc:511] Master 127.3.12.126:39893 requested a full tablet report, sending...
I20260805 23:05:24.094818 9175 ts_manager.cc:194] Registered new tserver with Master: 38f462034dc74d1e8320655b75b832d7 (127.3.12.67:33347)
I20260805 23:05:24.095394 9175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:44853
I20260805 23:05:24.097426 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:05:24.107117 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.099343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.3.12.126:39893)
0805 23:05:24.099488 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.099493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.099681 (+ 188us) server_negotiation.cc:407] Connection header received
0805 23:05:24.099846 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.099848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.099888 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.099973 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:24.100771 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.101279 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.102134 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.102353 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.102461 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.103847 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:24.103851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:24.103854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:24.103887 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:24.105355 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:24.105792 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:24.105796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:24.105798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:24.105883 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:24.106223 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:24.106227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:24.106229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:24.106447 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:24.106604 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.106828 (+ 224us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.106945 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:05:24.110034 9175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:24.113124 9175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:24.127588 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.122636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.3.12.65:43757)
0805 23:05:24.122939 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.122948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.122972 (+ 24us) server_negotiation.cc:407] Connection header received
0805 23:05:24.123050 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.123056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.123294 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.123459 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:24.123782 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.124806 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.126328 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.126891 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.127023 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.127200 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.127327 (+ 127us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.127428 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
I20260805 23:05:24.127568 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.121703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:43757 (local address 127.0.0.1:35454)
0805 23:05:24.122073 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.122146 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:24.122272 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:24.123444 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:24.123450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:24.123475 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:24.123625 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.123632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.124912 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.124917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.126197 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.126208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.127022 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.127025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.127149 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.127186 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 23:05:24.127298 (+ 112us) client_negotiation.cc:240] Negotiation successful
0805 23:05:24.127418 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260805 23:05:24.128703 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.122176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:33347 (local address 127.0.0.1:39694)
0805 23:05:24.123703 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.123716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:24.123801 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:24.124532 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:24.124535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:24.124554 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:24.124658 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.124664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.125765 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.125772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.127108 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.127120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.128238 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.128242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.128360 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.128386 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 23:05:24.128451 (+ 65us) client_negotiation.cc:240] Negotiation successful
0805 23:05:24.128524 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"mutex_wait_us":51,"thread_start_us":85,"threads_started":1}
I20260805 23:05:24.128752 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.123301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.3.12.67:33347)
0805 23:05:24.124054 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.124066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.124087 (+ 21us) server_negotiation.cc:407] Connection header received
0805 23:05:24.124158 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.124162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.124373 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.124531 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:24.124777 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.125638 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.127258 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.128114 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.128246 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.128430 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.128498 (+ 68us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.128591 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":140,"threads_started":1}
I20260805 23:05:24.128746 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.121914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:39043 (local address 127.0.0.1:45936)
0805 23:05:24.122550 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.122564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:24.122672 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:24.123697 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:24.123703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:24.123726 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:24.123828 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.123837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.125213 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.125216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.127110 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.127120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.128346 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.128350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.128453 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.128483 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 23:05:24.128533 (+ 50us) client_negotiation.cc:240] Negotiation successful
0805 23:05:24.128597 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":151,"thread_start_us":110,"threads_started":1}
I20260805 23:05:24.129590 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.122629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.3.12.66:39043)
0805 23:05:24.123007 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.123019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.123044 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:05:24.123245 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.123251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.123499 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.123703 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:24.123915 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.125076 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.127258 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.128187 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.129140 (+ 953us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.129276 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.129320 (+ 44us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.129406 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":146,"threads_started":1}
I20260805 23:05:24.130926 9350 tablet_service.cc:1467] Processing CreateTablet for tablet 28de3ec2c2054de6be8069c62a5d9a0e (DEFAULT_TABLE table=test-table [id=a1407d8b003c4826b67f8d97f35faaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:24.131569 9729 tablet_service.cc:1467] Processing CreateTablet for tablet 28de3ec2c2054de6be8069c62a5d9a0e (DEFAULT_TABLE table=test-table [id=a1407d8b003c4826b67f8d97f35faaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:24.132179 9350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de3ec2c2054de6be8069c62a5d9a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:24.132691 9729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de3ec2c2054de6be8069c62a5d9a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:24.134673 9538 tablet_service.cc:1467] Processing CreateTablet for tablet 28de3ec2c2054de6be8069c62a5d9a0e (DEFAULT_TABLE table=test-table [id=a1407d8b003c4826b67f8d97f35faaeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:24.135533 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de3ec2c2054de6be8069c62a5d9a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:24.140031 9857 tablet_bootstrap.cc:492] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: Bootstrap starting.
I20260805 23:05:24.140492 9856 tablet_bootstrap.cc:492] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1: Bootstrap starting.
I20260805 23:05:24.142796 9859 tablet_bootstrap.cc:492] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: Bootstrap starting.
I20260805 23:05:24.142930 9856 tablet_bootstrap.cc:654] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:24.143676 9856 log.cc:824] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:24.144943 9859 tablet_bootstrap.cc:654] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:24.145795 9857 tablet_bootstrap.cc:654] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:24.145859 9856 tablet_bootstrap.cc:492] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1: No bootstrap required, opened a new log
I20260805 23:05:24.146054 9856 ts_tablet_manager.cc:1397] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:05:24.146256 9859 log.cc:824] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:24.147075 9857 log.cc:824] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:24.148438 9859 tablet_bootstrap.cc:492] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: No bootstrap required, opened a new log
I20260805 23:05:24.148649 9859 ts_tablet_manager.cc:1397] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:05:24.149019 9857 tablet_bootstrap.cc:492] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: No bootstrap required, opened a new log
I20260805 23:05:24.149212 9857 ts_tablet_manager.cc:1397] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260805 23:05:24.150583 9856 raft_consensus.cc:348] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.150856 9856 raft_consensus.cc:374] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:24.150933 9856 raft_consensus.cc:729] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ca203f7270444688c9b574a4c903d1, State: Initialized, Role: FOLLOWER
I20260805 23:05:24.151448 9856 consensus_queue.cc:260] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.152184 9438 heartbeater.cc:503] Master 127.3.12.126:39893 was elected leader, sending a full tablet report...
I20260805 23:05:24.152364 9857 raft_consensus.cc:348] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.152597 9857 raft_consensus.cc:374] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:24.152657 9857 raft_consensus.cc:729] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f462034dc74d1e8320655b75b832d7, State: Initialized, Role: FOLLOWER
I20260805 23:05:24.152508 9859 raft_consensus.cc:348] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.152779 9859 raft_consensus.cc:374] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:24.152849 9859 raft_consensus.cc:729] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dac0353aa4441969b68ea7da157ccd0, State: Initialized, Role: FOLLOWER
I20260805 23:05:24.153153 9857 consensus_queue.cc:260] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.153265 9859 consensus_queue.cc:260] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.153769 9856 ts_tablet_manager.cc:1428] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260805 23:05:24.154089 9857 ts_tablet_manager.cc:1428] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:05:24.154143 9859 ts_tablet_manager.cc:1428] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:05:24.154402 9833 heartbeater.cc:503] Master 127.3.12.126:39893 was elected leader, sending a full tablet report...
I20260805 23:05:24.154448 9630 heartbeater.cc:503] Master 127.3.12.126:39893 was elected leader, sending a full tablet report...
I20260805 23:05:24.193758 9869 raft_consensus.cc:482] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:24.194026 9869 raft_consensus.cc:504] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.195528 9869 leader_election.cc:288] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93ca203f7270444688c9b574a4c903d1 (127.3.12.65:43757), 38f462034dc74d1e8320655b75b832d7 (127.3.12.67:33347)
I20260805 23:05:24.199811 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.195735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:43757 (local address 127.3.12.66:33841)
0805 23:05:24.196415 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.196427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:24.196524 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:24.196757 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:24.196760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:24.196780 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:24.196905 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.196910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.197683 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.197688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.198529 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.198546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.199370 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.199374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.199479 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.199500 (+ 21us) client_negotiation.cc:769] Sending connection context
0805 23:05:24.199603 (+ 103us) client_negotiation.cc:240] Negotiation successful
0805 23:05:24.199676 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603}
I20260805 23:05:24.199812 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.196055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:33841 (local address 127.3.12.65:43757)
0805 23:05:24.196269 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.196274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.196498 (+ 224us) server_negotiation.cc:407] Connection header received
0805 23:05:24.196598 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.196600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.196646 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.196743 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:24.197019 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.197569 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.198669 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.199251 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.199368 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.199432 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.199619 (+ 187us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.199689 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260805 23:05:24.199998 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.195848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:33347 (local address 127.3.12.66:58353)
0805 23:05:24.196720 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.196729 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:24.196837 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:24.197091 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:24.197094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:24.197109 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:24.197239 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.197245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.198024 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.198030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.198871 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:24.198881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.199558 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.199561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:24.199673 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.199697 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 23:05:24.199793 (+ 96us) client_negotiation.cc:240] Negotiation successful
0805 23:05:24.199861 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":163,"threads_started":1}
I20260805 23:05:24.199996 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.196095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:58353 (local address 127.3.12.67:33347)
0805 23:05:24.196223 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.196228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.196825 (+ 597us) server_negotiation.cc:407] Connection header received
0805 23:05:24.196931 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.196934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.196996 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.197080 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:24.197356 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.197909 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.199007 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.199429 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.199553 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.199613 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.199811 (+ 198us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.199877 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260805 23:05:24.200762 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de3ec2c2054de6be8069c62a5d9a0e" candidate_uuid: "2dac0353aa4441969b68ea7da157ccd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f462034dc74d1e8320655b75b832d7" is_pre_election: true
I20260805 23:05:24.200762 9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de3ec2c2054de6be8069c62a5d9a0e" candidate_uuid: "2dac0353aa4441969b68ea7da157ccd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ca203f7270444688c9b574a4c903d1" is_pre_election: true
I20260805 23:05:24.201090 9377 raft_consensus.cc:2454] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dac0353aa4441969b68ea7da157ccd0 in term 0.
I20260805 23:05:24.201090 9756 raft_consensus.cc:2454] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dac0353aa4441969b68ea7da157ccd0 in term 0.
I20260805 23:05:24.201643 9470 leader_election.cc:302] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dac0353aa4441969b68ea7da157ccd0, 38f462034dc74d1e8320655b75b832d7, 93ca203f7270444688c9b574a4c903d1; no voters:
I20260805 23:05:24.201912 9869 raft_consensus.cc:2781] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:24.202006 9869 raft_consensus.cc:482] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:24.202059 9869 raft_consensus.cc:3037] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:24.203235 9869 raft_consensus.cc:504] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.203576 9869 leader_election.cc:288] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [CANDIDATE]: Term 1 election: Requested vote from peers 93ca203f7270444688c9b574a4c903d1 (127.3.12.65:43757), 38f462034dc74d1e8320655b75b832d7 (127.3.12.67:33347)
I20260805 23:05:24.203912 9377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de3ec2c2054de6be8069c62a5d9a0e" candidate_uuid: "2dac0353aa4441969b68ea7da157ccd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ca203f7270444688c9b574a4c903d1"
I20260805 23:05:24.203987 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de3ec2c2054de6be8069c62a5d9a0e" candidate_uuid: "2dac0353aa4441969b68ea7da157ccd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f462034dc74d1e8320655b75b832d7"
I20260805 23:05:24.204037 9377 raft_consensus.cc:3037] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:24.204152 9756 raft_consensus.cc:3037] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:24.205124 9377 raft_consensus.cc:2454] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dac0353aa4441969b68ea7da157ccd0 in term 1.
I20260805 23:05:24.205335 9756 raft_consensus.cc:2454] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dac0353aa4441969b68ea7da157ccd0 in term 1.
I20260805 23:05:24.205467 9470 leader_election.cc:302] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dac0353aa4441969b68ea7da157ccd0, 93ca203f7270444688c9b574a4c903d1; no voters:
I20260805 23:05:24.205650 9869 raft_consensus.cc:2781] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:24.205935 9869 raft_consensus.cc:686] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 1 LEADER]: Becoming Leader. State: Replica: 2dac0353aa4441969b68ea7da157ccd0, State: Running, Role: LEADER
I20260805 23:05:24.206290 9869 consensus_queue.cc:237] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } }
I20260805 23:05:24.209647 9175 catalog_manager.cc:5495] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dac0353aa4441969b68ea7da157ccd0 (127.3.12.66). New cstate: current_term: 1 leader_uuid: "2dac0353aa4441969b68ea7da157ccd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2dac0353aa4441969b68ea7da157ccd0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 39043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:24.229905 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.226442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.3.12.66:39043)
0805 23:05:24.226576 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.226580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.226593 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:05:24.226686 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.226688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.226745 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.226828 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:24.227078 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.227586 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.228474 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:24.228693 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:24.228771 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:24.229045 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:24.229454 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:24.229572 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.229636 (+ 64us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.229689 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:05:24.235915 9377 raft_consensus.cc:1261] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [term 1 FOLLOWER]: Refusing update from remote peer 2dac0353aa4441969b68ea7da157ccd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:24.235914 9756 raft_consensus.cc:1261] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 1 FOLLOWER]: Refusing update from remote peer 2dac0353aa4441969b68ea7da157ccd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:24.236730 9869 consensus_queue.cc:1035] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f462034dc74d1e8320655b75b832d7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:24.236958 9871 consensus_queue.cc:1035] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ca203f7270444688c9b574a4c903d1" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 43757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:24.254115 9169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:24.254274 9169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:24.256731 9169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfe234c007a4b3f95fa6e48ee7d261a: Sending DeleteTablet for 3 replicas of tablet 28de3ec2c2054de6be8069c62a5d9a0e
I20260805 23:05:24.257642 9350 tablet_service.cc:1514] Processing DeleteTablet for tablet 28de3ec2c2054de6be8069c62a5d9a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:24 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:35454
I20260805 23:05:24.257747 9729 tablet_service.cc:1514] Processing DeleteTablet for tablet 28de3ec2c2054de6be8069c62a5d9a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:24 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:39694
I20260805 23:05:24.257750 9538 tablet_service.cc:1514] Processing DeleteTablet for tablet 28de3ec2c2054de6be8069c62a5d9a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:24 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:45936
I20260805 23:05:24.258226 9889 tablet_replica.cc:330] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: stopping tablet replica
I20260805 23:05:24.258366 9888 tablet_replica.cc:330] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1: stopping tablet replica
I20260805 23:05:24.258543 9889 raft_consensus.cc:2229] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:24.258662 9888 raft_consensus.cc:2229] T 28de3ec2c2054de6be8069c62a5d9a0e P 93ca203f7270444688c9b574a4c903d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:24.258785 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 9256
I20260805 23:05:24.280406 9890 tablet_replica.cc:330] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: stopping tablet replica
I20260805 23:05:24.280894 9890 raft_consensus.cc:2229] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:24.281354 9890 raft_consensus.cc:2258] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:24.283461 9889 raft_consensus.cc:2258] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:24.285027 9890 ts_tablet_manager.cc:1905] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:24.285175 9889 ts_tablet_manager.cc:1905] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:24.288076 9890 ts_tablet_manager.cc:1918] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:24.288321 9890 log.cc:1196] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/28de3ec2c2054de6be8069c62a5d9a0e
I20260805 23:05:24.288787 9890 ts_tablet_manager.cc:1939] T 28de3ec2c2054de6be8069c62a5d9a0e P 38f462034dc74d1e8320655b75b832d7: Deleting consensus metadata
I20260805 23:05:24.288820 9889 ts_tablet_manager.cc:1918] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:24.288934 9889 log.cc:1196] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/28de3ec2c2054de6be8069c62a5d9a0e
I20260805 23:05:24.289583 9889 ts_tablet_manager.cc:1939] T 28de3ec2c2054de6be8069c62a5d9a0e P 2dac0353aa4441969b68ea7da157ccd0: Deleting consensus metadata
I20260805 23:05:24.290511 9150 catalog_manager.cc:4841] TS 38f462034dc74d1e8320655b75b832d7 (127.3.12.67:33347): tablet 28de3ec2c2054de6be8069c62a5d9a0e (table test-table [id=a1407d8b003c4826b67f8d97f35faaeb]) successfully deleted
I20260805 23:05:24.290838 9153 catalog_manager.cc:4841] TS 2dac0353aa4441969b68ea7da157ccd0 (127.3.12.66:39043): tablet 28de3ec2c2054de6be8069c62a5d9a0e (table test-table [id=a1407d8b003c4826b67f8d97f35faaeb]) successfully deleted
W20260805 23:05:24.291234 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:43757 (error 108)
I20260805 23:05:24.292428 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.292059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:43757 (local address 127.0.0.1:35462)
0805 23:05:24.292214 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.292325 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:43757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260805 23:05:24.292518 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 9445
W20260805 23:05:24.292636 9155 catalog_manager.cc:4584] TS 93ca203f7270444688c9b574a4c903d1 (127.3.12.65:43757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28de3ec2c2054de6be8069c62a5d9a0e: Network error: Client connection negotiation failed: client connection to 127.3.12.65:43757: connect: Connection refused (error 111)
I20260805 23:05:24.305692 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 9639
I20260805 23:05:24.314491 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 9129
2026-08-05T23:05:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2577 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-05T23:05:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:24Z Disabled control of system clock
I20260805 23:05:24.354858 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:46511
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:46511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260805 23:05:24.470256 9897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:24.470547 9897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:24.470587 9897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:24.474620 9897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:24.474677 9897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:24.474694 9897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:24.474710 9897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:24.474725 9897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:24.479385 9897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:46511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:46511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:24.480567 9897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:24.481982 9897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:24.490636 9904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:24.490689 9906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:24.490675 9908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:24.490867 9897 server_base.cc:1034] running on GCE node
I20260805 23:05:24.491617 9897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:24.492894 9897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:24.494089 9897 hybrid_clock.cc:648] HybridClock initialized: now 1785971124494073 us; error 34 us; skew 500 ppm
I20260805 23:05:24.496068 9897 webserver.cc:460] Webserver started at http://127.3.12.126:40963/ using document root <none> and password file <none>
I20260805 23:05:24.496742 9897 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:24.496840 9897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:24.497061 9897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:24.499135 9897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "06cacb0959034063b0f4bbe749167cc9"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "ef039737afe29bf8007f3e004d6aebe7"
server_key_iv: "6af3fdd5937af4f130d0c437606aa238"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.499877 9897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "06cacb0959034063b0f4bbe749167cc9"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "ef039737afe29bf8007f3e004d6aebe7"
server_key_iv: "6af3fdd5937af4f130d0c437606aa238"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.503964 9897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:05:24.506387 9916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.507543 9897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260805 23:05:24.507682 9897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "06cacb0959034063b0f4bbe749167cc9"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "ef039737afe29bf8007f3e004d6aebe7"
server_key_iv: "6af3fdd5937af4f130d0c437606aa238"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.507804 9897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:24.512423 9897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:24.513036 9897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:24.513222 9897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:24.520336 9897 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:46511
I20260805 23:05:24.520362 9994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:46511 every 8 connection(s)
I20260805 23:05:24.521397 9897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:24.525032 9995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:24.530817 9995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9: Bootstrap starting.
I20260805 23:05:24.530892 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 9897
I20260805 23:05:24.531020 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:24.531308 3121 external_mini_cluster.cc:1428] Setting key c529bd1d85c8b1d22a55142a6740c1cd
I20260805 23:05:24.533557 9995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:24.534313 9995 log.cc:824] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:24.536688 9995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9: No bootstrap required, opened a new log
I20260805 23:05:24.537427 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.532857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.3.12.126:46511)
0805 23:05:24.533522 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.533539 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.533575 (+ 36us) server_negotiation.cc:407] Connection header received
0805 23:05:24.534290 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.534308 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.534725 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.535017 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:24.535848 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:24.535854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:24.535869 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:24.535882 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:24.535957 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:24.535980 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:24.536109 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.536181 (+ 72us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.536480 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":154,"threads_started":1}
I20260805 23:05:24.540947 9995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cacb0959034063b0f4bbe749167cc9" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 46511 } }
I20260805 23:05:24.541167 9995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:24.541211 9995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cacb0959034063b0f4bbe749167cc9, State: Initialized, Role: FOLLOWER
I20260805 23:05:24.541571 9995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cacb0959034063b0f4bbe749167cc9" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 46511 } }
I20260805 23:05:24.541697 9995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:24.541742 9995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:24.541811 9995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:24.542697 9995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cacb0959034063b0f4bbe749167cc9" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 46511 } }
I20260805 23:05:24.543028 9995 leader_election.cc:302] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06cacb0959034063b0f4bbe749167cc9; no voters:
I20260805 23:05:24.543345 9995 leader_election.cc:288] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:24.543437 10002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:24.543687 10002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 06cacb0959034063b0f4bbe749167cc9, State: Running, Role: LEADER
I20260805 23:05:24.543989 10002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cacb0959034063b0f4bbe749167cc9" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 46511 } }
I20260805 23:05:24.544737 9995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:24.545872 10003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06cacb0959034063b0f4bbe749167cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cacb0959034063b0f4bbe749167cc9" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 46511 } } }
I20260805 23:05:24.545917 10004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06cacb0959034063b0f4bbe749167cc9. Latest consensus state: current_term: 1 leader_uuid: "06cacb0959034063b0f4bbe749167cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cacb0959034063b0f4bbe749167cc9" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 46511 } } }
I20260805 23:05:24.546007 10003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:24.546022 10004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:24.546413 10015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:24.549453 10015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:24.555011 10015 catalog_manager.cc:1347] Generated new cluster ID: f5ac13c07ac94ca98349c5be07faebbf
I20260805 23:05:24.555081 10015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:24.565550 10015 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:24.565873 10015 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:24.583216 10015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9: Generated new TSK 0
I20260805 23:05:24.584098 10015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:05:24.592545 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:46511
--builtin_ntp_servers=127.3.12.84:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260805 23:05:24.709699 10030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:24.709942 10030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:24.709981 10030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:24.714217 10030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:24.714280 10030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:24.714335 10030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:24.719185 10030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:24.720198 10030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:24.721614 10030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:24.731407 10040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:24.731402 10039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:24.731402 10043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:24.731822 10030 server_base.cc:1034] running on GCE node
I20260805 23:05:24.732232 10030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:24.732925 10030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:24.734093 10030 hybrid_clock.cc:648] HybridClock initialized: now 1785971124734076 us; error 25 us; skew 500 ppm
I20260805 23:05:24.735934 10030 webserver.cc:460] Webserver started at http://127.3.12.65:46729/ using document root <none> and password file <none>
I20260805 23:05:24.736584 10030 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:24.736658 10030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:24.736877 10030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:24.738929 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "8bddbfebe7bc4fc5abf67649595e12b5"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "480281dd4123b1a20e9c52003d86a7fe"
server_key_iv: "124543435c67699d0bab0dac0c3b9e2f"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.739528 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "8bddbfebe7bc4fc5abf67649595e12b5"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "480281dd4123b1a20e9c52003d86a7fe"
server_key_iv: "124543435c67699d0bab0dac0c3b9e2f"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.743295 10030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260805 23:05:24.746049 10049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.747365 10030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 23:05:24.747504 10030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "8bddbfebe7bc4fc5abf67649595e12b5"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "480281dd4123b1a20e9c52003d86a7fe"
server_key_iv: "124543435c67699d0bab0dac0c3b9e2f"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.747625 10030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:24.752230 10030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:24.752918 10030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:24.753104 10030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:24.753662 10030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:24.754563 10030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:24.754642 10030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.754705 10030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:24.754746 10030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.763685 10030 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:42205
I20260805 23:05:24.763707 10219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:42205 every 8 connection(s)
I20260805 23:05:24.764585 10030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:24.769874 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 10030
I20260805 23:05:24.770015 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:24.770334 3121 external_mini_cluster.cc:1428] Setting key 6228abf76b099b8824b6782a17ac8dd4
I20260805 23:05:24.770907 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.766433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:46775 (local address 127.3.12.126:46511)
0805 23:05:24.766585 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.766589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.767638 (+ 1049us) server_negotiation.cc:407] Connection header received
0805 23:05:24.769067 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.769069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.769114 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.769192 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:24.770395 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:24.770398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:24.770400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:24.770428 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:24.770453 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:24.770465 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:24.770577 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.770727 (+ 150us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.770772 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260805 23:05:24.772176 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.766838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:46511 (local address 127.3.12.65:46775)
0805 23:05:24.767492 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.767531 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:24.768869 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:24.769350 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:24.769357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:24.769440 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:24.770124 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:05:24.770125 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:24.770165 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:05:24.770231 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:24.770563 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:24.770573 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:24.770579 (+ 6us) client_negotiation.cc:769] Sending connection context
0805 23:05:24.770765 (+ 186us) client_negotiation.cc:240] Negotiation successful
0805 23:05:24.771010 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260805 23:05:24.772326 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:46511
--builtin_ntp_servers=127.3.12.84:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260805 23:05:24.773411 10221 heartbeater.cc:347] Connected to a master server at 127.3.12.126:46511
I20260805 23:05:24.773658 10221 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:24.774129 10221 heartbeater.cc:511] Master 127.3.12.126:46511 requested a full tablet report, sending...
I20260805 23:05:24.775938 9947 ts_manager.cc:194] Registered new tserver with Master: 8bddbfebe7bc4fc5abf67649595e12b5 (127.3.12.65:42205)
W20260805 23:05:24.884295 10224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:24.884554 10224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:24.884594 10224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:24.888677 10224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:24.888731 10224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:24.888784 10224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:24.893455 10224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:24.894441 10224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:24.895793 10224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:24.905750 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:24.905777 10238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:24.905834 10224 server_base.cc:1034] running on GCE node
W20260805 23:05:24.905777 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:24.906476 10224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:24.907032 10224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:24.908232 10224 hybrid_clock.cc:648] HybridClock initialized: now 1785971124908208 us; error 34 us; skew 500 ppm
I20260805 23:05:24.910104 10224 webserver.cc:460] Webserver started at http://127.3.12.66:46219/ using document root <none> and password file <none>
I20260805 23:05:24.910686 10224 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:24.910781 10224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:24.911043 10224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:24.913244 10224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "067d881f7dbd41baa965917c0915a339"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "1bb363bb3fe529f46afb4c432ef58fb4"
server_key_iv: "ea3d2a7c0d334d57c72e924f9e3f6b36"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.913820 10224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "067d881f7dbd41baa965917c0915a339"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "1bb363bb3fe529f46afb4c432ef58fb4"
server_key_iv: "ea3d2a7c0d334d57c72e924f9e3f6b36"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.917441 10224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:05:24.919831 10246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.921149 10224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:24.921288 10224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "067d881f7dbd41baa965917c0915a339"
format_stamp: "Formatted at 2026-08-05 23:05:24 on dist-test-slave-2d19"
server_key: "1bb363bb3fe529f46afb4c432ef58fb4"
server_key_iv: "ea3d2a7c0d334d57c72e924f9e3f6b36"
server_key_version: "encryptionkey@0"
I20260805 23:05:24.921401 10224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:24.926203 10224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:24.926803 10224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:24.926994 10224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:24.927654 10224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:24.928679 10224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:24.928750 10224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.928813 10224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:24.928860 10224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:24.938342 10224 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:45311
I20260805 23:05:24.938364 10407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:45311 every 8 connection(s)
I20260805 23:05:24.939306 10224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:24.940128 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 10224
I20260805 23:05:24.940258 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:24.940629 3121 external_mini_cluster.cc:1428] Setting key 3199499115cf03de40d1666904dfa59e
I20260805 23:05:24.942891 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:46511
--builtin_ntp_servers=127.3.12.84:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260805 23:05:24.946412 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.941394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:58591 (local address 127.3.12.126:46511)
0805 23:05:24.941542 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:24.941545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:24.942859 (+ 1314us) server_negotiation.cc:407] Connection header received
0805 23:05:24.944338 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:24.944345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:24.944411 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:24.944519 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:24.945865 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:24.945869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:24.945872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:24.945885 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:24.945914 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:24.945923 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:24.946025 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:24.946216 (+ 191us) server_negotiation.cc:299] Negotiation successful
0805 23:05:24.946263 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:05:24.947356 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:24.941915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:46511 (local address 127.3.12.66:58591)
0805 23:05:24.942682 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:24.942735 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:24.944025 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:24.944705 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:24.944711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:24.944804 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:24.945520 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:05:24.945522 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:24.945566 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:05:24.945639 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:24.946053 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:24.946062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:24.946068 (+ 6us) client_negotiation.cc:769] Sending connection context
0805 23:05:24.946259 (+ 191us) client_negotiation.cc:240] Negotiation successful
0805 23:05:24.946506 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":140,"threads_started":1}
I20260805 23:05:24.948645 10410 heartbeater.cc:347] Connected to a master server at 127.3.12.126:46511
I20260805 23:05:24.948889 10410 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:24.949357 10410 heartbeater.cc:511] Master 127.3.12.126:46511 requested a full tablet report, sending...
I20260805 23:05:24.950536 9947 ts_manager.cc:194] Registered new tserver with Master: 067d881f7dbd41baa965917c0915a339 (127.3.12.66:45311)
W20260805 23:05:25.058327 10417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:25.058576 10417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:25.058615 10417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:25.062548 10417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:25.062602 10417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:25.062655 10417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:25.067193 10417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:25.068213 10417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:25.069623 10417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:25.079193 10425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:25.079195 10424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:25.079205 10427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:25.079483 10417 server_base.cc:1034] running on GCE node
I20260805 23:05:25.079888 10417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:25.080544 10417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:25.081760 10417 hybrid_clock.cc:648] HybridClock initialized: now 1785971125081734 us; error 53 us; skew 500 ppm
I20260805 23:05:25.083801 10417 webserver.cc:460] Webserver started at http://127.3.12.67:45181/ using document root <none> and password file <none>
I20260805 23:05:25.084434 10417 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:25.084525 10417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:25.084769 10417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:25.086849 10417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "6cbb329c11914cafa3cd2287fc3dc3f8"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "740630f527c03340708d83cd7cc433fc"
server_key_iv: "59b36fb9c5658872a7f23efb95cb9989"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.087463 10417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "6cbb329c11914cafa3cd2287fc3dc3f8"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "740630f527c03340708d83cd7cc433fc"
server_key_iv: "59b36fb9c5658872a7f23efb95cb9989"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.091234 10417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260805 23:05:25.093779 10435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:25.095098 10417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:25.095232 10417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "6cbb329c11914cafa3cd2287fc3dc3f8"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "740630f527c03340708d83cd7cc433fc"
server_key_iv: "59b36fb9c5658872a7f23efb95cb9989"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.095351 10417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:25.099974 10417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:25.100668 10417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:25.100847 10417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:25.101383 10417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:25.102334 10417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:25.102402 10417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:25.102471 10417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:25.102523 10417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:25.111712 10417 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:37365
I20260805 23:05:25.111734 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:37365 every 8 connection(s)
I20260805 23:05:25.112706 10417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:25.119194 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.114464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:46881 (local address 127.3.12.126:46511)
0805 23:05:25.114610 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.114613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.115609 (+ 996us) server_negotiation.cc:407] Connection header received
0805 23:05:25.117265 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.117269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.117317 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.117433 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.118638 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.118642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.118644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.118665 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.118684 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.118693 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.118798 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.119006 (+ 208us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.119058 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:05:25.120030 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.114807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:46511 (local address 127.3.12.67:46881)
0805 23:05:25.115451 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.115500 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:25.117045 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:25.117609 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:25.117615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:25.117701 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:25.118310 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:05:25.118312 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:25.118352 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:05:25.118444 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:25.118782 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:25.118792 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:25.118798 (+ 6us) client_negotiation.cc:769] Sending connection context
0805 23:05:25.118984 (+ 186us) client_negotiation.cc:240] Negotiation successful
0805 23:05:25.119229 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260805 23:05:25.121183 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 10417
I20260805 23:05:25.121308 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:25.121392 10592 heartbeater.cc:347] Connected to a master server at 127.3.12.126:46511
I20260805 23:05:25.121636 3121 external_mini_cluster.cc:1428] Setting key 5e2c1adf0dea196a5aa7a9e756ee19d6
I20260805 23:05:25.121654 10592 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:25.122067 10592 heartbeater.cc:511] Master 127.3.12.126:46511 requested a full tablet report, sending...
I20260805 23:05:25.123236 9947 ts_manager.cc:194] Registered new tserver with Master: 6cbb329c11914cafa3cd2287fc3dc3f8 (127.3.12.67:37365)
I20260805 23:05:25.123976 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:05:25.127770 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.126164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.3.12.126:46511)
0805 23:05:25.126298 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.126303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.126458 (+ 155us) server_negotiation.cc:407] Connection header received
0805 23:05:25.126631 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.126634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.126669 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.126757 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.127382 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.127385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.127388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.127400 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.127420 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.127429 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.127534 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.127607 (+ 73us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.127648 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:05:25.130668 9944 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:25.133780 9944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:25.145581 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.142050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42205 (local address 127.0.0.1:59982)
0805 23:05:25.142757 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.142807 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:25.142904 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:25.143804 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:25.143810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:25.143886 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:25.144695 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:05:25.144696 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:25.144717 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:05:25.144779 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:25.145193 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:25.145203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:25.145210 (+ 7us) client_negotiation.cc:769] Sending connection context
0805 23:05:25.145304 (+ 94us) client_negotiation.cc:240] Negotiation successful
0805 23:05:25.145410 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":80,"threads_started":1}
I20260805 23:05:25.145601 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.142848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.3.12.66:45311)
0805 23:05:25.143676 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.143684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.143702 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:05:25.143749 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.143753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.143970 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.144146 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.144937 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.144947 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.144961 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.144975 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.145018 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.145065 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.145194 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.145370 (+ 176us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.145445 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":141,"threads_started":1}
I20260805 23:05:25.145581 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.142147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:45311 (local address 127.0.0.1:34358)
0805 23:05:25.143111 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.143123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:25.143198 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:25.144155 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:25.144159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:25.144175 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:25.144695 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:05:25.144697 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:25.144718 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:05:25.144784 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:25.145173 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:25.145212 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:25.145216 (+ 4us) client_negotiation.cc:769] Sending connection context
0805 23:05:25.145339 (+ 123us) client_negotiation.cc:240] Negotiation successful
0805 23:05:25.145424 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":20,"thread_start_us":83,"threads_started":1}
I20260805 23:05:25.146323 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.142193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.3.12.65:42205)
0805 23:05:25.143426 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.143435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.143455 (+ 20us) server_negotiation.cc:407] Connection header received
0805 23:05:25.143506 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.143510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.143655 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.143809 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.144919 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.144925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.144937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.144949 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.145023 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.145060 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.146032 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.146084 (+ 52us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.146154 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":105,"threads_started":1}
I20260805 23:05:25.149039 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.142807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:37365 (local address 127.0.0.1:41976)
0805 23:05:25.144908 (+ 2101us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.144922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:25.145004 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:25.146689 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:25.146692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:25.146706 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:25.148367 (+ 1661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:05:25.148368 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:25.148377 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:05:25.148401 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:25.148798 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:25.148801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:25.148804 (+ 3us) client_negotiation.cc:769] Sending connection context
0805 23:05:25.148880 (+ 76us) client_negotiation.cc:240] Negotiation successful
0805 23:05:25.148917 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2037,"thread_start_us":108,"threads_started":1}
I20260805 23:05:25.149513 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.143258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.3.12.67:37365)
0805 23:05:25.143562 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.143574 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.145031 (+ 1457us) server_negotiation.cc:407] Connection header received
0805 23:05:25.145098 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.145107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.145876 (+ 769us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.146035 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.148531 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.148537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.148545 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.148557 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.148593 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.148613 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.148749 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.149306 (+ 557us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.149381 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":119,"threads_started":1}
I20260805 23:05:25.149312 10121 tablet_service.cc:1467] Processing CreateTablet for tablet 1183fd2230ab4ca39adbcb7da094b117 (DEFAULT_TABLE table=test-table [id=f56527839ce2420f9d51a6f1d9b19f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:25.149312 10320 tablet_service.cc:1467] Processing CreateTablet for tablet 1183fd2230ab4ca39adbcb7da094b117 (DEFAULT_TABLE table=test-table [id=f56527839ce2420f9d51a6f1d9b19f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:25.150487 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1183fd2230ab4ca39adbcb7da094b117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:25.150488 10121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1183fd2230ab4ca39adbcb7da094b117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:25.152182 10495 tablet_service.cc:1467] Processing CreateTablet for tablet 1183fd2230ab4ca39adbcb7da094b117 (DEFAULT_TABLE table=test-table [id=f56527839ce2420f9d51a6f1d9b19f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:25.153340 10495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1183fd2230ab4ca39adbcb7da094b117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:25.158329 10614 tablet_bootstrap.cc:492] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: Bootstrap starting.
I20260805 23:05:25.158756 10616 tablet_bootstrap.cc:492] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5: Bootstrap starting.
I20260805 23:05:25.159503 10617 tablet_bootstrap.cc:492] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: Bootstrap starting.
I20260805 23:05:25.161185 10616 tablet_bootstrap.cc:654] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:25.161305 10617 tablet_bootstrap.cc:654] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:25.162003 10617 log.cc:824] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:25.162062 10616 log.cc:824] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:25.162354 10614 tablet_bootstrap.cc:654] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:25.163142 10614 log.cc:824] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:25.164187 10616 tablet_bootstrap.cc:492] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5: No bootstrap required, opened a new log
I20260805 23:05:25.164187 10617 tablet_bootstrap.cc:492] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: No bootstrap required, opened a new log
I20260805 23:05:25.164402 10617 ts_tablet_manager.cc:1397] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20260805 23:05:25.164402 10616 ts_tablet_manager.cc:1397] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:05:25.165225 10614 tablet_bootstrap.cc:492] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: No bootstrap required, opened a new log
I20260805 23:05:25.165429 10614 ts_tablet_manager.cc:1397] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260805 23:05:25.168715 10614 raft_consensus.cc:348] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.168962 10614 raft_consensus.cc:374] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:25.169013 10614 raft_consensus.cc:729] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067d881f7dbd41baa965917c0915a339, State: Initialized, Role: FOLLOWER
I20260805 23:05:25.169054 10617 raft_consensus.cc:348] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.169054 10616 raft_consensus.cc:348] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.169310 10617 raft_consensus.cc:374] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:25.169310 10616 raft_consensus.cc:374] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:25.169368 10617 raft_consensus.cc:729] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbb329c11914cafa3cd2287fc3dc3f8, State: Initialized, Role: FOLLOWER
I20260805 23:05:25.169366 10616 raft_consensus.cc:729] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bddbfebe7bc4fc5abf67649595e12b5, State: Initialized, Role: FOLLOWER
I20260805 23:05:25.169507 10614 consensus_queue.cc:260] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.169814 10617 consensus_queue.cc:260] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.169814 10616 consensus_queue.cc:260] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.170796 10617 ts_tablet_manager.cc:1428] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:05:25.170785 10616 ts_tablet_manager.cc:1428] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 23:05:25.170811 10614 ts_tablet_manager.cc:1428] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260805 23:05:25.171185 10221 heartbeater.cc:503] Master 127.3.12.126:46511 was elected leader, sending a full tablet report...
I20260805 23:05:25.171190 10592 heartbeater.cc:503] Master 127.3.12.126:46511 was elected leader, sending a full tablet report...
I20260805 23:05:25.172384 10410 heartbeater.cc:503] Master 127.3.12.126:46511 was elected leader, sending a full tablet report...
W20260805 23:05:25.190279 10413 tablet.cc:2369] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:05:25.196139 10625 raft_consensus.cc:482] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:25.196424 10625 raft_consensus.cc:504] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.197686 10625 leader_election.cc:288] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 067d881f7dbd41baa965917c0915a339 (127.3.12.66:45311), 6cbb329c11914cafa3cd2287fc3dc3f8 (127.3.12.67:37365)
I20260805 23:05:25.200314 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.197930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:37365 (local address 127.3.12.65:57881)
0805 23:05:25.198290 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.198307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:25.198395 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:25.199070 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:25.199075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:25.199091 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:25.199642 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:05:25.199644 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:25.199658 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:05:25.199695 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:25.200006 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:25.200028 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:25.200035 (+ 7us) client_negotiation.cc:769] Sending connection context
0805 23:05:25.200123 (+ 88us) client_negotiation.cc:240] Negotiation successful
0805 23:05:25.200161 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261}
I20260805 23:05:25.200371 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.198183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:57881 (local address 127.3.12.67:37365)
0805 23:05:25.198529 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.198533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.198550 (+ 17us) server_negotiation.cc:407] Connection header received
0805 23:05:25.198616 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.198619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.198660 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.198776 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.199832 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.199838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.199842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.199860 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.199886 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.199896 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.200016 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.200127 (+ 111us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.200171 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260805 23:05:25.201004 10523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1183fd2230ab4ca39adbcb7da094b117" candidate_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" is_pre_election: true
I20260805 23:05:25.201370 10523 raft_consensus.cc:2454] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bddbfebe7bc4fc5abf67649595e12b5 in term 0.
I20260805 23:05:25.201565 10612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.198130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:42779 (local address 127.3.12.66:45311)
0805 23:05:25.198360 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.198366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.198784 (+ 418us) server_negotiation.cc:407] Connection header received
0805 23:05:25.198903 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.198909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.198998 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.199138 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.201071 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.201074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.201077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.201090 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.201133 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.201148 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.201244 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.201397 (+ 153us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.201450 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260805 23:05:25.201581 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.197930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:45311 (local address 127.3.12.65:42779)
0805 23:05:25.198654 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.198668 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:25.198764 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:25.199184 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:25.199189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:25.199207 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:25.200888 (+ 1681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:05:25.200890 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:25.200900 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:05:25.200927 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:25.201291 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:25.201297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:25.201302 (+ 5us) client_negotiation.cc:769] Sending connection context
0805 23:05:25.201384 (+ 82us) client_negotiation.cc:240] Negotiation successful
0805 23:05:25.201421 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":115,"thread_start_us":147,"threads_started":1}
I20260805 23:05:25.201932 10050 leader_election.cc:302] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cbb329c11914cafa3cd2287fc3dc3f8, 8bddbfebe7bc4fc5abf67649595e12b5; no voters:
I20260805 23:05:25.202225 10625 raft_consensus.cc:2781] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:25.202342 10625 raft_consensus.cc:482] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:25.202400 10625 raft_consensus.cc:3037] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:25.202728 10345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1183fd2230ab4ca39adbcb7da094b117" candidate_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067d881f7dbd41baa965917c0915a339" is_pre_election: true
I20260805 23:05:25.203013 10345 raft_consensus.cc:2454] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bddbfebe7bc4fc5abf67649595e12b5 in term 0.
I20260805 23:05:25.203738 10625 raft_consensus.cc:504] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.204093 10625 leader_election.cc:288] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [CANDIDATE]: Term 1 election: Requested vote from peers 067d881f7dbd41baa965917c0915a339 (127.3.12.66:45311), 6cbb329c11914cafa3cd2287fc3dc3f8 (127.3.12.67:37365)
I20260805 23:05:25.204447 10345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1183fd2230ab4ca39adbcb7da094b117" candidate_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067d881f7dbd41baa965917c0915a339"
I20260805 23:05:25.204557 10345 raft_consensus.cc:3037] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:25.205533 10345 raft_consensus.cc:2454] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bddbfebe7bc4fc5abf67649595e12b5 in term 1.
I20260805 23:05:25.205838 10053 leader_election.cc:302] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067d881f7dbd41baa965917c0915a339, 8bddbfebe7bc4fc5abf67649595e12b5; no voters:
I20260805 23:05:25.205821 10523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1183fd2230ab4ca39adbcb7da094b117" candidate_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8"
I20260805 23:05:25.205953 10523 raft_consensus.cc:3037] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:25.206020 10625 raft_consensus.cc:2781] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:25.206251 10625 raft_consensus.cc:686] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 1 LEADER]: Becoming Leader. State: Replica: 8bddbfebe7bc4fc5abf67649595e12b5, State: Running, Role: LEADER
I20260805 23:05:25.206604 10625 consensus_queue.cc:237] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } }
I20260805 23:05:25.207273 10523 raft_consensus.cc:2454] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bddbfebe7bc4fc5abf67649595e12b5 in term 1.
I20260805 23:05:25.210096 9944 catalog_manager.cc:5495] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bddbfebe7bc4fc5abf67649595e12b5 (127.3.12.65). New cstate: current_term: 1 leader_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bddbfebe7bc4fc5abf67649595e12b5" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:25.223219 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.221847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.3.12.65:42205)
0805 23:05:25.221970 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.221973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.221987 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:25.222097 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.222099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.222155 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.222226 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.222925 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.222927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.222929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.222937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.222952 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.222958 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.223025 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.223076 (+ 51us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.223111 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260805 23:05:25.229359 10345 raft_consensus.cc:1261] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 1 FOLLOWER]: Refusing update from remote peer 8bddbfebe7bc4fc5abf67649595e12b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:25.229359 10523 raft_consensus.cc:1261] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 1 FOLLOWER]: Refusing update from remote peer 8bddbfebe7bc4fc5abf67649595e12b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:25.230103 10625 consensus_queue.cc:1035] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cbb329c11914cafa3cd2287fc3dc3f8" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:25.230367 10627 consensus_queue.cc:1035] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067d881f7dbd41baa965917c0915a339" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:25.244956 10633 mvcc.cc:205] Tried to move back new op lower bound from 7315337728932114432 to 7315337728849448960. Current Snapshot: MvccSnapshot[applied={T|T < 7315337728932114432}]
I20260805 23:05:25.250295 9944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:25.250499 9944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:25.253177 9944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06cacb0959034063b0f4bbe749167cc9: Sending DeleteTablet for 3 replicas of tablet 1183fd2230ab4ca39adbcb7da094b117
I20260805 23:05:25.253948 10121 tablet_service.cc:1514] Processing DeleteTablet for tablet 1183fd2230ab4ca39adbcb7da094b117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:25 UTC) from {username='slave'} at 127.0.0.1:59982
I20260805 23:05:25.253948 10320 tablet_service.cc:1514] Processing DeleteTablet for tablet 1183fd2230ab4ca39adbcb7da094b117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:25 UTC) from {username='slave'} at 127.0.0.1:34358
I20260805 23:05:25.254453 10645 tablet_replica.cc:330] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5: stopping tablet replica
I20260805 23:05:25.254604 10495 tablet_service.cc:1514] Processing DeleteTablet for tablet 1183fd2230ab4ca39adbcb7da094b117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:25 UTC) from {username='slave'} at 127.0.0.1:41976
I20260805 23:05:25.254779 10645 raft_consensus.cc:2229] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:25.255035 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 10030
I20260805 23:05:25.255123 10645 raft_consensus.cc:2258] T 1183fd2230ab4ca39adbcb7da094b117 P 8bddbfebe7bc4fc5abf67649595e12b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:25.255290 10648 tablet_replica.cc:330] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: stopping tablet replica
I20260805 23:05:25.256067 10648 raft_consensus.cc:2229] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:25.256563 10648 raft_consensus.cc:2258] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:25.259240 10648 ts_tablet_manager.cc:1905] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:25.262023 10648 ts_tablet_manager.cc:1918] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:25.262118 10648 log.cc:1196] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/1183fd2230ab4ca39adbcb7da094b117
I20260805 23:05:25.262470 10648 ts_tablet_manager.cc:1939] T 1183fd2230ab4ca39adbcb7da094b117 P 6cbb329c11914cafa3cd2287fc3dc3f8: Deleting consensus metadata
I20260805 23:05:25.263675 9917 catalog_manager.cc:4841] TS 6cbb329c11914cafa3cd2287fc3dc3f8 (127.3.12.67:37365): tablet 1183fd2230ab4ca39adbcb7da094b117 (table test-table [id=f56527839ce2420f9d51a6f1d9b19f4f]) successfully deleted
I20260805 23:05:25.264523 10646 tablet_replica.cc:330] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: stopping tablet replica
I20260805 23:05:25.264832 10646 raft_consensus.cc:2229] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:25.265129 10646 raft_consensus.cc:2258] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:25.266469 10646 ts_tablet_manager.cc:1905] T 1183fd2230ab4ca39adbcb7da094b117 P 067d881f7dbd41baa965917c0915a339: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260805 23:05:25.266976 9919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:42205 (error 108)
I20260805 23:05:25.267057 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 10224
I20260805 23:05:25.274278 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.273957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42205 (local address 127.0.0.1:60000)
0805 23:05:25.274113 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.274196 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:42205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260805 23:05:25.274454 9919 catalog_manager.cc:4584] TS 8bddbfebe7bc4fc5abf67649595e12b5 (127.3.12.65:42205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1183fd2230ab4ca39adbcb7da094b117: Network error: Client connection negotiation failed: client connection to 127.3.12.65:42205: connect: Connection refused (error 111)
I20260805 23:05:25.278973 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 10417
I20260805 23:05:25.285075 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.279008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:45311 (local address 127.0.0.1:34370)
0805 23:05:25.284900 (+ 5892us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.284969 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.66:45311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5779}
I20260805 23:05:25.292332 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 9897
2026-08-05T23:05:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (974 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260805 23:05:25.320060 3121 mini_oidc.cc:152] Starting JWKS server
I20260805 23:05:25.321837 3121 webserver.cc:460] Webserver started at https://127.0.0.1:40539/ using document root <none> and password file <none>
I20260805 23:05:25.322033 3121 mini_oidc.cc:189] Starting OIDC Discovery server
I20260805 23:05:25.322324 3121 webserver.cc:460] Webserver started at http://127.0.0.1:40191/ using document root <none> and password file <none>
2026-08-05T23:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:25Z Disabled control of system clock
I20260805 23:05:25.332883 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36377
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:35911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:36377
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40539/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/testchainca.cert with env {}
W20260805 23:05:25.446667 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:25.446957 10657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:25.447000 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:25.450986 10657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:25.451048 10657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:25.451064 10657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:25.451086 10657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:25.451102 10657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260805 23:05:25.451117 10657 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260805 23:05:25.451131 10657 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40539/jwks.json
I20260805 23:05:25.456501 10657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:36377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40539/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:25.457567 10657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:25.458670 10657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:25.467319 10670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:25.467350 10666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:25.467312 10665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:25.467499 10657 server_base.cc:1034] running on GCE node
I20260805 23:05:25.468176 10657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:25.469305 10657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:25.470479 10657 hybrid_clock.cc:648] HybridClock initialized: now 1785971125470451 us; error 42 us; skew 500 ppm
I20260805 23:05:25.472440 10657 webserver.cc:460] Webserver started at http://127.3.12.126:42899/ using document root <none> and password file <none>
I20260805 23:05:25.473023 10657 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:25.473093 10657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:25.473263 10657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:25.475286 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "6dd71af8607944cea607c1b054ed1d52"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "123b93530e081e4ebc332d13440e7cd6"
server_key_iv: "6d9cf469420dc073597ec156f83b843b"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.475818 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "6dd71af8607944cea607c1b054ed1d52"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "123b93530e081e4ebc332d13440e7cd6"
server_key_iv: "6d9cf469420dc073597ec156f83b843b"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.479429 10657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260805 23:05:25.481834 10677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:25.483125 10657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:25.483227 10657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "6dd71af8607944cea607c1b054ed1d52"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "123b93530e081e4ebc332d13440e7cd6"
server_key_iv: "6d9cf469420dc073597ec156f83b843b"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.483312 10657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:25.511336 10657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:25.512063 10657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:25.512245 10657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:25.519847 10657 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:36377
I20260805 23:05:25.519860 10745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:36377 every 8 connection(s)
I20260805 23:05:25.520968 10657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:25.524670 10746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:25.529449 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 10657
I20260805 23:05:25.529600 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:25.529878 3121 external_mini_cluster.cc:1428] Setting key 3811b97924223464961907396e2456fc
I20260805 23:05:25.531482 10746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52: Bootstrap starting.
I20260805 23:05:25.534034 10746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:25.534842 10746 log.cc:824] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:25.536935 10746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52: No bootstrap required, opened a new log
I20260805 23:05:25.539721 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.531386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.3.12.126:36377)
0805 23:05:25.532114 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.532131 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.532170 (+ 39us) server_negotiation.cc:407] Connection header received
0805 23:05:25.532938 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.532958 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.533400 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.533761 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.534278 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.535311 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.536242 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.536768 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.536948 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:25.537869 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.537874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.537888 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.538299 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.538360 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.538382 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.538501 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.538675 (+ 174us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.538954 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260805 23:05:25.541117 10746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd71af8607944cea607c1b054ed1d52" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36377 } }
I20260805 23:05:25.541424 10746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:25.541493 10746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd71af8607944cea607c1b054ed1d52, State: Initialized, Role: FOLLOWER
I20260805 23:05:25.541962 10746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd71af8607944cea607c1b054ed1d52" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36377 } }
I20260805 23:05:25.542133 10746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:25.542200 10746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:25.542310 10746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:25.543061 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:05:25.543517 10746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd71af8607944cea607c1b054ed1d52" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36377 } }
I20260805 23:05:25.543879 10746 leader_election.cc:302] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dd71af8607944cea607c1b054ed1d52; no voters:
I20260805 23:05:25.544241 10746 leader_election.cc:288] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:25.544360 10752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:25.544556 10752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [term 1 LEADER]: Becoming Leader. State: Replica: 6dd71af8607944cea607c1b054ed1d52, State: Running, Role: LEADER
I20260805 23:05:25.544888 10752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd71af8607944cea607c1b054ed1d52" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36377 } }
I20260805 23:05:25.545668 10746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:25.546221 10752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dd71af8607944cea607c1b054ed1d52. Latest consensus state: current_term: 1 leader_uuid: "6dd71af8607944cea607c1b054ed1d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd71af8607944cea607c1b054ed1d52" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36377 } } }
I20260805 23:05:25.546350 10752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:25.546672 10753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dd71af8607944cea607c1b054ed1d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd71af8607944cea607c1b054ed1d52" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36377 } } }
I20260805 23:05:25.546813 10753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52 [sys.catalog]: This master's current role is: LEADER
W20260805 23:05:25.551182 10773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:05:25.551296 10773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:05:25.551378 10756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:25.552182 10756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:25.558403 10756 catalog_manager.cc:1347] Generated new cluster ID: c1abe1e74fe34d6c9a50e6fc65c80916
I20260805 23:05:25.558487 10756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:25.574282 10756 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:25.575246 10756 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:25.581895 10756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dd71af8607944cea607c1b054ed1d52: Generated new TSK 0
I20260805 23:05:25.582615 10756 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260805 23:05:25.615613 10788 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260805 23:05:25.615770 10788 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260805 23:05:25.616400 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.598196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.3.12.126:36377)
0805 23:05:25.598346 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.598351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.598606 (+ 255us) server_negotiation.cc:407] Connection header received
0805 23:05:25.598827 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.598831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.598894 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.599029 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0805 23:05:25.599334 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.600190 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.601152 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.601513 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.601623 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:25.602000 (+ 377us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0805 23:05:25.615921 (+ 13921us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0805 23:05:25.616217 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260805 23:05:25.616478 10749 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260805 23:05:25.617738 10787 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:25.598140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:36377 (local address 127.0.0.1:39744)
0805 23:05:25.598520 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.598534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:25.598660 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:25.599084 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:25.599087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:25.599095 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0805 23:05:25.599214 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:25.599219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.600376 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.600380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:25.601028 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:25.601036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.601638 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.601641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:25.601712 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:25.601784 (+ 72us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0805 23:05:25.616173 (+ 14389us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0805 23:05:25.616494 (+ 321us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.12.126:36377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":159,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.12.126:36377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260805 23:05:25.618922 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 10657
2026-08-05T23:05:25Z chronyd exiting
I20260805 23:05:25.668418 3121 test_util.cc:175] -----------------------------------------------
I20260805 23:05:25.668514 3121 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785971095475883-3121-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (350 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260805 23:05:25.669513 3121 mini_oidc.cc:152] Starting JWKS server
I20260805 23:05:25.670334 3121 webserver.cc:460] Webserver started at https://127.0.0.1:38543/ using document root <none> and password file <none>
I20260805 23:05:25.670409 3121 mini_oidc.cc:189] Starting OIDC Discovery server
I20260805 23:05:25.670660 3121 webserver.cc:460] Webserver started at http://127.0.0.1:35065/ using document root <none> and password file <none>
2026-08-05T23:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:25Z Disabled control of system clock
I20260805 23:05:25.681111 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:39687
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:37065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:39687
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38543/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/testchainca.cert with env {}
W20260805 23:05:25.793776 10796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:25.794031 10796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:25.794071 10796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:25.798053 10796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:25.798110 10796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:25.798126 10796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:25.798143 10796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:25.798158 10796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260805 23:05:25.798173 10796 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260805 23:05:25.798187 10796 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38543/jwks.json
I20260805 23:05:25.803341 10796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:37065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:39687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38543/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:25.804396 10796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:25.805366 10796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:25.813990 10807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:25.814044 10805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:25.814057 10804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:25.814186 10796 server_base.cc:1034] running on GCE node
I20260805 23:05:25.814749 10796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:25.815913 10796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:25.817113 10796 hybrid_clock.cc:648] HybridClock initialized: now 1785971125817078 us; error 47 us; skew 500 ppm
I20260805 23:05:25.818938 10796 webserver.cc:460] Webserver started at http://127.3.12.126:41273/ using document root <none> and password file <none>
I20260805 23:05:25.819538 10796 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:25.819638 10796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:25.819876 10796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:25.822016 10796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "8ca376ec6356451b8e811d1e82cbadfb"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "b7708630b01fe1354b5051aa646dd032"
server_key_iv: "6e145e011bf04b210d891c431c08cb2a"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.822616 10796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "8ca376ec6356451b8e811d1e82cbadfb"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "b7708630b01fe1354b5051aa646dd032"
server_key_iv: "6e145e011bf04b210d891c431c08cb2a"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.826529 10796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:05:25.829255 10816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:25.830482 10796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260805 23:05:25.830628 10796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "8ca376ec6356451b8e811d1e82cbadfb"
format_stamp: "Formatted at 2026-08-05 23:05:25 on dist-test-slave-2d19"
server_key: "b7708630b01fe1354b5051aa646dd032"
server_key_iv: "6e145e011bf04b210d891c431c08cb2a"
server_key_version: "encryptionkey@0"
I20260805 23:05:25.830746 10796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:25.849287 10796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:25.850046 10796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:25.850266 10796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:25.857631 10796 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:39687
I20260805 23:05:25.857652 10895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:39687 every 8 connection(s)
I20260805 23:05:25.858603 10796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:25.861999 10896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:25.867594 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 10796
I20260805 23:05:25.867630 10896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb: Bootstrap starting.
I20260805 23:05:25.867731 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:25.868027 3121 external_mini_cluster.cc:1428] Setting key 9d5aac1a9a35cb1f617a7b804e47fa18
I20260805 23:05:25.870419 10896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:25.871270 10896 log.cc:824] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:25.873281 10896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb: No bootstrap required, opened a new log
I20260805 23:05:25.877588 10896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca376ec6356451b8e811d1e82cbadfb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39687 } }
I20260805 23:05:25.877795 10896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:25.877827 10896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca376ec6356451b8e811d1e82cbadfb, State: Initialized, Role: FOLLOWER
I20260805 23:05:25.877867 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.869465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.3.12.126:39687)
0805 23:05:25.870173 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.870190 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.870224 (+ 34us) server_negotiation.cc:407] Connection header received
0805 23:05:25.870963 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.870983 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.871436 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.871797 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:25.872243 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.873248 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.874479 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.874953 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.875073 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:25.875967 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:25.875972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:25.875985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:25.876443 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:25.876500 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:25.876521 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:25.876607 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:25.876793 (+ 186us) server_negotiation.cc:299] Negotiation successful
0805 23:05:25.877091 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":145,"threads_started":1}
I20260805 23:05:25.878361 10896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca376ec6356451b8e811d1e82cbadfb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39687 } }
I20260805 23:05:25.878484 10896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:25.878566 10896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:25.878681 10896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:25.879766 10896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca376ec6356451b8e811d1e82cbadfb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39687 } }
I20260805 23:05:25.880147 10896 leader_election.cc:302] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ca376ec6356451b8e811d1e82cbadfb; no voters:
I20260805 23:05:25.880540 10896 leader_election.cc:288] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:25.880679 10905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:25.880934 10905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [term 1 LEADER]: Becoming Leader. State: Replica: 8ca376ec6356451b8e811d1e82cbadfb, State: Running, Role: LEADER
I20260805 23:05:25.880990 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:05:25.881242 10905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca376ec6356451b8e811d1e82cbadfb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39687 } }
I20260805 23:05:25.881692 10896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:25.882485 10909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca376ec6356451b8e811d1e82cbadfb. Latest consensus state: current_term: 1 leader_uuid: "8ca376ec6356451b8e811d1e82cbadfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca376ec6356451b8e811d1e82cbadfb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39687 } } }
I20260805 23:05:25.882637 10909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:25.883029 10916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:25.883283 10908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ca376ec6356451b8e811d1e82cbadfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca376ec6356451b8e811d1e82cbadfb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39687 } } }
I20260805 23:05:25.883417 10908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:25.886063 10916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:25.892086 10916 catalog_manager.cc:1347] Generated new cluster ID: 1c725d9fcebb41a391c5bf8c852b34b2
I20260805 23:05:25.892175 10916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:25.909765 10916 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:25.911053 10916 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:25.927496 10916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ca376ec6356451b8e811d1e82cbadfb: Generated new TSK 0
I20260805 23:05:25.928540 10916 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260805 23:05:25.960366 10944 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260805 23:05:25.960531 10944 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260805 23:05:25.961339 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:25.945029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.3.12.126:39687)
0805 23:05:25.945234 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:25.945240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:25.945376 (+ 136us) server_negotiation.cc:407] Connection header received
0805 23:05:25.945520 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:25.945523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:25.945588 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:25.945714 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0805 23:05:25.945929 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.946775 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.947627 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.947987 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.948085 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:25.948453 (+ 368us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0805 23:05:25.960877 (+ 12424us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0805 23:05:25.961174 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":94}
W20260805 23:05:25.961410 10902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260805 23:05:25.961971 10941 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:25.944891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:39687 (local address 127.0.0.1:36002)
0805 23:05:25.945287 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:25.945300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:25.945400 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:25.945716 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:25.945718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:25.945725 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0805 23:05:25.945821 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:25.945825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.946883 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.946885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:25.947511 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:25.947518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:25.948129 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:25.948132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:25.948202 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:25.948225 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0805 23:05:25.961214 (+ 12989us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0805 23:05:25.961466 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.12.126:39687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":145,"threads_started":1}
I20260805 23:05:25.962929 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 10796
2026-08-05T23:05:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (346 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260805 23:05:26.016209 3121 mini_oidc.cc:152] Starting JWKS server
I20260805 23:05:26.017251 3121 webserver.cc:460] Webserver started at https://127.0.0.1:41067/ using document root <none> and password file <none>
I20260805 23:05:26.017400 3121 mini_oidc.cc:189] Starting OIDC Discovery server
I20260805 23:05:26.017697 3121 webserver.cc:460] Webserver started at http://127.0.0.1:38335/ using document root <none> and password file <none>
2026-08-05T23:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:26Z Disabled control of system clock
I20260805 23:05:26.029047 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:33285
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:33285
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41067/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260805 23:05:26.151698 10950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:26.151991 10950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:26.152034 10950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:26.156152 10950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:26.156215 10950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:26.156231 10950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:26.156248 10950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:26.156328 10950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260805 23:05:26.156352 10950 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260805 23:05:26.156366 10950 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41067/jwks.json
I20260805 23:05:26.161183 10950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:33285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41067/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:26.162196 10950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:26.163585 10950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:26.172470 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:26.172465 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:26.172519 10950 server_base.cc:1034] running on GCE node
W20260805 23:05:26.172465 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:26.173305 10950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:26.174512 10950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:26.175717 10950 hybrid_clock.cc:648] HybridClock initialized: now 1785971126175683 us; error 49 us; skew 500 ppm
I20260805 23:05:26.177763 10950 webserver.cc:460] Webserver started at http://127.3.12.126:40119/ using document root <none> and password file <none>
I20260805 23:05:26.178376 10950 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:26.178442 10950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:26.178702 10950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:26.180922 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "b840585d716244de966587b4886da982"
format_stamp: "Formatted at 2026-08-05 23:05:26 on dist-test-slave-2d19"
server_key: "6a51fa67943f6f35d10cc1c3f9318e63"
server_key_iv: "599b901e51bcf6a56255dd16f00f8a03"
server_key_version: "encryptionkey@0"
I20260805 23:05:26.181553 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "b840585d716244de966587b4886da982"
format_stamp: "Formatted at 2026-08-05 23:05:26 on dist-test-slave-2d19"
server_key: "6a51fa67943f6f35d10cc1c3f9318e63"
server_key_iv: "599b901e51bcf6a56255dd16f00f8a03"
server_key_version: "encryptionkey@0"
I20260805 23:05:26.185475 10950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260805 23:05:26.187955 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:26.189353 10950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:05:26.189507 10950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "b840585d716244de966587b4886da982"
format_stamp: "Formatted at 2026-08-05 23:05:26 on dist-test-slave-2d19"
server_key: "6a51fa67943f6f35d10cc1c3f9318e63"
server_key_iv: "599b901e51bcf6a56255dd16f00f8a03"
server_key_version: "encryptionkey@0"
I20260805 23:05:26.189627 10950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:26.206805 10950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:26.207527 10950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:26.207731 10950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:26.214751 10950 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:33285
I20260805 23:05:26.214759 11051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:33285 every 8 connection(s)
I20260805 23:05:26.215796 10950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:26.219184 11052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:26.224826 11052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982: Bootstrap starting.
I20260805 23:05:26.226012 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 10950
I20260805 23:05:26.226163 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:26.226459 3121 external_mini_cluster.cc:1428] Setting key 407bd04dbe15451ffb26ebe9d31ba449
I20260805 23:05:26.227671 11052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:26.228591 11052 log.cc:824] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:26.230813 11052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982: No bootstrap required, opened a new log
I20260805 23:05:26.235236 11052 raft_consensus.cc:348] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b840585d716244de966587b4886da982" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33285 } }
I20260805 23:05:26.235495 11052 raft_consensus.cc:374] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:26.235576 11052 raft_consensus.cc:729] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b840585d716244de966587b4886da982, State: Initialized, Role: FOLLOWER
I20260805 23:05:26.236056 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:26.227639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.3.12.126:33285)
0805 23:05:26.228411 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:26.228434 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:26.228486 (+ 52us) server_negotiation.cc:407] Connection header received
0805 23:05:26.229192 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:26.229210 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:26.229551 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:26.229908 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:26.230267 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.231279 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.232529 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.233005 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.233126 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:26.234085 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:26.234089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:26.234103 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:26.234497 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:26.234569 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:26.234591 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:26.234674 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:26.234843 (+ 169us) server_negotiation.cc:299] Negotiation successful
0805 23:05:26.235184 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":167,"threads_started":1}
I20260805 23:05:26.236071 11052 consensus_queue.cc:260] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b840585d716244de966587b4886da982" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33285 } }
I20260805 23:05:26.236303 11052 raft_consensus.cc:388] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:26.236385 11052 raft_consensus.cc:482] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:26.236487 11052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:26.237586 11052 raft_consensus.cc:504] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b840585d716244de966587b4886da982" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33285 } }
I20260805 23:05:26.237963 11052 leader_election.cc:302] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b840585d716244de966587b4886da982; no voters:
I20260805 23:05:26.238322 11052 leader_election.cc:288] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:26.238466 11057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:26.238734 11057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [term 1 LEADER]: Becoming Leader. State: Replica: b840585d716244de966587b4886da982, State: Running, Role: LEADER
I20260805 23:05:26.239070 11057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b840585d716244de966587b4886da982" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33285 } }
I20260805 23:05:26.239300 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:05:26.239703 11052 sys_catalog.cc:564] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:26.240366 11057 sys_catalog.cc:455] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b840585d716244de966587b4886da982. Latest consensus state: current_term: 1 leader_uuid: "b840585d716244de966587b4886da982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b840585d716244de966587b4886da982" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33285 } } }
I20260805 23:05:26.240494 11057 sys_catalog.cc:458] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:26.240811 11059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b840585d716244de966587b4886da982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b840585d716244de966587b4886da982" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33285 } } }
I20260805 23:05:26.240918 11059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:26.241197 11064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:26.244428 11064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:26.249941 11064 catalog_manager.cc:1347] Generated new cluster ID: 6f9dc675a877482e87bce4dc24208991
I20260805 23:05:26.250020 11064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:26.258808 11064 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:26.259701 11064 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:26.271082 11064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b840585d716244de966587b4886da982: Generated new TSK 0
I20260805 23:05:26.271828 11064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:05:26.294946 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:26.289796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.3.12.126:33285)
0805 23:05:26.289918 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:26.289923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:26.290079 (+ 156us) server_negotiation.cc:407] Connection header received
0805 23:05:26.290215 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:26.290218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:26.290274 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:26.290391 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0805 23:05:26.290603 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.291435 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.292302 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.292679 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.292778 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:26.293105 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0805 23:05:26.294567 (+ 1462us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0805 23:05:26.294801 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260805 23:05:26.295037 11055 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260805 23:05:26.296634 11089 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:26.289645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:33285 (local address 127.0.0.1:39124)
0805 23:05:26.289993 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:26.290006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:26.290101 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:26.290395 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:26.290397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:26.290402 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0805 23:05:26.290501 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:26.290505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.291543 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.291545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:26.292168 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:26.292175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.292794 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.292797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:26.292859 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:26.292881 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0805 23:05:26.294727 (+ 1846us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0805 23:05:26.296464 (+ 1737us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.12.126:33285: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1554,"thread_start_us":117,"threads_started":1}
I20260805 23:05:26.297348 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 10950
E20260805 23:05:26.308384 11091 webserver.cc:533] Webserver: sslize failed:
E20260805 23:05:26.308476 11091 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-05T23:05:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-05T23:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:26Z Disabled control of system clock
I20260805 23:05:26.352344 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43545
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:41039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:43545
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/testchainca.cert with env {}
W20260805 23:05:26.465261 11096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:26.465584 11096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:26.465634 11096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:26.469779 11096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:26.469842 11096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:26.469859 11096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:26.469877 11096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:26.469892 11096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260805 23:05:26.469907 11096 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260805 23:05:26.469920 11096 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260805 23:05:26.475296 11096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:41039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:43545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:26.476327 11096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:26.477289 11096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:26.486033 11110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:26.486179 11113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:26.486024 11111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:26.486315 11096 server_base.cc:1034] running on GCE node
I20260805 23:05:26.486954 11096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:26.488104 11096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:26.489305 11096 hybrid_clock.cc:648] HybridClock initialized: now 1785971126489277 us; error 46 us; skew 500 ppm
I20260805 23:05:26.497430 11096 webserver.cc:460] Webserver started at http://127.3.12.126:42649/ using document root <none> and password file <none>
I20260805 23:05:26.498104 11096 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:26.498207 11096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:26.498456 11096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:26.500687 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "0b313ab9179a40a9a6699445f5507d83"
format_stamp: "Formatted at 2026-08-05 23:05:26 on dist-test-slave-2d19"
server_key: "591f19ef831ae61293db0ba7c3d360c2"
server_key_iv: "3ef19ef84556e264935ec2a8e6585b19"
server_key_version: "encryptionkey@0"
I20260805 23:05:26.501287 11096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "0b313ab9179a40a9a6699445f5507d83"
format_stamp: "Formatted at 2026-08-05 23:05:26 on dist-test-slave-2d19"
server_key: "591f19ef831ae61293db0ba7c3d360c2"
server_key_iv: "3ef19ef84556e264935ec2a8e6585b19"
server_key_version: "encryptionkey@0"
I20260805 23:05:26.505120 11096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 23:05:26.507609 11123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:26.508848 11096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:26.508999 11096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "0b313ab9179a40a9a6699445f5507d83"
format_stamp: "Formatted at 2026-08-05 23:05:26 on dist-test-slave-2d19"
server_key: "591f19ef831ae61293db0ba7c3d360c2"
server_key_iv: "3ef19ef84556e264935ec2a8e6585b19"
server_key_version: "encryptionkey@0"
I20260805 23:05:26.509120 11096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:26.544792 11096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:26.545596 11096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:26.545819 11096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:26.553591 11096 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:43545
I20260805 23:05:26.553594 11199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:43545 every 8 connection(s)
I20260805 23:05:26.554677 11096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:26.558284 11200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:26.559062 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 11096
I20260805 23:05:26.559226 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:26.559526 3121 external_mini_cluster.cc:1428] Setting key 733533c5a930cc38b9f1218de9f94ae8
I20260805 23:05:26.564640 11200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83: Bootstrap starting.
I20260805 23:05:26.567369 11200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:26.568292 11200 log.cc:824] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:26.569191 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:26.561029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.3.12.126:43545)
0805 23:05:26.561675 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:26.561691 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:26.561727 (+ 36us) server_negotiation.cc:407] Connection header received
0805 23:05:26.562353 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:26.562370 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:26.562790 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:26.563066 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:26.563437 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.564461 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.565640 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.566111 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.566228 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:26.567102 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:26.567107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:26.567119 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:05:26.567561 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:26.567621 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:05:26.567642 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:26.567721 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:26.567889 (+ 168us) server_negotiation.cc:299] Negotiation successful
0805 23:05:26.568208 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":133,"threads_started":1}
I20260805 23:05:26.570488 11200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83: No bootstrap required, opened a new log
I20260805 23:05:26.572345 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:05:26.575410 11200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b313ab9179a40a9a6699445f5507d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43545 } }
I20260805 23:05:26.575734 11200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:26.575809 11200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b313ab9179a40a9a6699445f5507d83, State: Initialized, Role: FOLLOWER
I20260805 23:05:26.576381 11200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b313ab9179a40a9a6699445f5507d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43545 } }
I20260805 23:05:26.576576 11200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:26.576674 11200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:26.576804 11200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:26.578303 11200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b313ab9179a40a9a6699445f5507d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43545 } }
I20260805 23:05:26.578727 11200 leader_election.cc:302] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b313ab9179a40a9a6699445f5507d83; no voters:
I20260805 23:05:26.579106 11200 leader_election.cc:288] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:26.579232 11205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:26.579481 11205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [term 1 LEADER]: Becoming Leader. State: Replica: 0b313ab9179a40a9a6699445f5507d83, State: Running, Role: LEADER
I20260805 23:05:26.579821 11205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b313ab9179a40a9a6699445f5507d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43545 } }
I20260805 23:05:26.580559 11200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:26.583024 11206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b313ab9179a40a9a6699445f5507d83. Latest consensus state: current_term: 1 leader_uuid: "0b313ab9179a40a9a6699445f5507d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b313ab9179a40a9a6699445f5507d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43545 } } }
I20260805 23:05:26.583160 11206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:26.583557 11207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b313ab9179a40a9a6699445f5507d83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b313ab9179a40a9a6699445f5507d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43545 } } }
I20260805 23:05:26.583685 11207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:26.583541 11216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:26.587224 11216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:26.597543 11216 catalog_manager.cc:1347] Generated new cluster ID: e722067339cc48afa2d114b9b1adcc3a
I20260805 23:05:26.597633 11216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:26.616189 11216 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:26.617733 11216 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:26.630340 11216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b313ab9179a40a9a6699445f5507d83: Generated new TSK 0
I20260805 23:05:26.631317 11216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:05:26.683555 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:26.643094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.3.12.126:43545)
0805 23:05:26.643251 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:26.643256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:26.643390 (+ 134us) server_negotiation.cc:407] Connection header received
0805 23:05:26.643530 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:26.643533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:26.643598 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:26.643732 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0805 23:05:26.643962 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.644842 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.645703 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.646071 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.646179 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:26.646460 (+ 281us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0805 23:05:26.683063 (+ 36603us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0805 23:05:26.683396 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43540: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260805 23:05:26.683645 11203 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43540: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260805 23:05:26.683697 11235 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:26.642928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43545 (local address 127.0.0.1:43540)
0805 23:05:26.643296 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:26.643310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:26.643412 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:26.643734 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:26.643736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:26.643742 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0805 23:05:26.643847 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:26.643851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.644953 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.644956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:26.645584 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:26.645592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:26.646187 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:26.646190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:26.646253 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:26.646277 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0805 23:05:26.683313 (+ 37036us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0805 23:05:26.683500 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.12.126:43545: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260805 23:05:26.684548 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 11096
2026-08-05T23:05:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (368 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (132 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:27 dist-test-slave-2d19 krb5kdc[11259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:27 dist-test-slave-2d19 krb5kdc[11259](info): set up 2 sockets
Aug 05 23:05:27 dist-test-slave-2d19 krb5kdc[11259](info): commencing operation
krb5kdc: starting...
W20260805 23:05:28.181955 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:28 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:28.317623 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36707
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:36707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:28.435015 11277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:28.435287 11277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:28.435360 11277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:28.439404 11277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:28.439471 11277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:28.439491 11277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:28.439553 11277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:28.439581 11277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:28.444913 11277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:36707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:28.445991 11277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:28.447057 11277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:28.455755 11287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:28.455765 11283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:28.455765 11282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:28.455962 11277 server_base.cc:1034] running on GCE node
I20260805 23:05:28.456674 11277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:28.457798 11277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:28.458989 11277 hybrid_clock.cc:648] HybridClock initialized: now 1785971128458961 us; error 41 us; skew 500 ppm
Aug 05 23:05:28 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971128, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:28.461815 11277 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:28.462719 11277 webserver.cc:460] Webserver started at http://127.3.12.126:33829/ using document root <none> and password file <none>
I20260805 23:05:28.463299 11277 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:28.463392 11277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:28.463634 11277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:28.465801 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "c21547438bfc4c7cb7713e5a11383090"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "7d5eda59a9e52041d1c889a833e064ad"
server_key_iv: "e99ddba63985c69a5d6eede468b92e83"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.466431 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "c21547438bfc4c7cb7713e5a11383090"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "7d5eda59a9e52041d1c889a833e064ad"
server_key_iv: "e99ddba63985c69a5d6eede468b92e83"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.470346 11277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:05:28.472954 11294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:28.474314 11277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:28.474465 11277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "c21547438bfc4c7cb7713e5a11383090"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "7d5eda59a9e52041d1c889a833e064ad"
server_key_iv: "e99ddba63985c69a5d6eede468b92e83"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.474587 11277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:28.494418 11277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:28.497479 11277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:28.497697 11277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:28.504973 11277 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:36707
I20260805 23:05:28.505035 11375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:36707 every 8 connection(s)
I20260805 23:05:28.505996 11277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:28.508867 11377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:28.514319 11377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090: Bootstrap starting.
I20260805 23:05:28.515075 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 11277
I20260805 23:05:28.515208 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:28.515530 3121 external_mini_cluster.cc:1428] Setting key 5774f07383cf0a6bfbe2a38219ca4e87
I20260805 23:05:28.517482 11377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:28.518297 11377 log.cc:824] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:28.520797 11377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090: No bootstrap required, opened a new log
Aug 05 23:05:28 dist-test-slave-2d19 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:28.525583 11377 raft_consensus.cc:348] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21547438bfc4c7cb7713e5a11383090" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36707 } }
I20260805 23:05:28.525852 11377 raft_consensus.cc:374] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:28.525933 11377 raft_consensus.cc:729] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c21547438bfc4c7cb7713e5a11383090, State: Initialized, Role: FOLLOWER
I20260805 23:05:28.526438 11377 consensus_queue.cc:260] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21547438bfc4c7cb7713e5a11383090" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36707 } }
I20260805 23:05:28.526604 11377 raft_consensus.cc:388] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:28.526695 11377 raft_consensus.cc:482] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:28.526799 11377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:28.527760 11377 raft_consensus.cc:504] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21547438bfc4c7cb7713e5a11383090" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36707 } }
I20260805 23:05:28.528162 11377 leader_election.cc:302] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c21547438bfc4c7cb7713e5a11383090; no voters:
I20260805 23:05:28.528573 11377 leader_election.cc:288] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:28.528733 11383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:28.528975 11383 raft_consensus.cc:686] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [term 1 LEADER]: Becoming Leader. State: Replica: c21547438bfc4c7cb7713e5a11383090, State: Running, Role: LEADER
I20260805 23:05:28.529294 11383 consensus_queue.cc:237] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21547438bfc4c7cb7713e5a11383090" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36707 } }
I20260805 23:05:28.530022 11377 sys_catalog.cc:564] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:28.530608 11383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c21547438bfc4c7cb7713e5a11383090. Latest consensus state: current_term: 1 leader_uuid: "c21547438bfc4c7cb7713e5a11383090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21547438bfc4c7cb7713e5a11383090" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36707 } } }
I20260805 23:05:28.530736 11383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:28.531176 11385 sys_catalog.cc:455] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c21547438bfc4c7cb7713e5a11383090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c21547438bfc4c7cb7713e5a11383090" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36707 } } }
I20260805 23:05:28.531283 11385 sys_catalog.cc:458] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:28.531528 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:28.516944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.3.12.126:36707)
0805 23:05:28.517411 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:28.517427 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:28.517465 (+ 38us) server_negotiation.cc:407] Connection header received
0805 23:05:28.518199 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:28.518218 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:28.518640 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:28.519065 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:28.519971 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:28.521051 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:28.522194 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:28.522671 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:28.522792 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:28.525875 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:28.525884 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:28.525904 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:28.525941 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:28.527841 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:28.528439 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:28.528444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:28.528449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:28.528606 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:28.528964 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:28.528967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:28.528969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:28.529558 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:28.529765 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:28.530093 (+ 328us) server_negotiation.cc:299] Negotiation successful
0805 23:05:28.530419 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":163,"threads_started":1}
W20260805 23:05:28.535414 11401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:05:28.535506 11401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:05:28.535626 11390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:28.536429 11390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:28.542752 11390 catalog_manager.cc:1347] Generated new cluster ID: bef4e883a8d941da8c714cb69e1abc98
I20260805 23:05:28.542834 11390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:28.554306 11390 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:28.555310 11390 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:28.559983 11390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c21547438bfc4c7cb7713e5a11383090: Generated new TSK 0
I20260805 23:05:28.560757 11390 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:05:28.596154 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36707
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:28.711066 11413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:28.711396 11413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:28.711488 11413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:28.715607 11413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:28.715720 11413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:28.715819 11413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:28.721261 11413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:28.722419 11413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:28.723543 11413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:28.733867 11421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:28.733894 11424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:28.733871 11422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:28.734252 11413 server_base.cc:1034] running on GCE node
I20260805 23:05:28.734745 11413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:28.735365 11413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:28.736541 11413 hybrid_clock.cc:648] HybridClock initialized: now 1785971128736504 us; error 46 us; skew 500 ppm
Aug 05 23:05:28 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971128, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:28.739521 11413 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:05:28.740599 11413 webserver.cc:460] Webserver started at http://127.3.12.65:44515/ using document root <none> and password file <none>
I20260805 23:05:28.741189 11413 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:28.741283 11413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:28.741540 11413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:28.743737 11413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "2fcdc4eed58f46359665319a5241ba30"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "5ed22899e850ee05d15f7e52618712f1"
server_key_iv: "3ebab7f1244842ae2eac70cd6bf70e28"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.744391 11413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "2fcdc4eed58f46359665319a5241ba30"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "5ed22899e850ee05d15f7e52618712f1"
server_key_iv: "3ebab7f1244842ae2eac70cd6bf70e28"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.748394 11413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260805 23:05:28.751057 11432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:28.752372 11413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:28.752506 11413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "2fcdc4eed58f46359665319a5241ba30"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "5ed22899e850ee05d15f7e52618712f1"
server_key_iv: "3ebab7f1244842ae2eac70cd6bf70e28"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.752632 11413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:28.765413 11413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:28.768369 11413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:28.768576 11413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:28.769169 11413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:28.770010 11413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:28.770081 11413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:28.770157 11413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:28.770210 11413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:28.780500 11413 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:34397
I20260805 23:05:28.780527 11603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:34397 every 8 connection(s)
I20260805 23:05:28.781432 11413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:28.783843 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 11413
I20260805 23:05:28.783994 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:28.784377 3121 external_mini_cluster.cc:1428] Setting key 74f802b3c27ac42ffb7554784bad38db
Aug 05 23:05:28 dist-test-slave-2d19 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971128, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:28.797205 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:28.783358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:47227 (local address 127.3.12.126:36707)
0805 23:05:28.783502 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:28.783505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:28.784619 (+ 1114us) server_negotiation.cc:407] Connection header received
0805 23:05:28.786363 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:28.786366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:28.786425 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:28.786512 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:28.787688 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:28.788233 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:28.790117 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:28.790355 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:28.790443 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:28.793376 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:28.793379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:28.793381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:28.793405 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:28.794996 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:28.795547 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:28.795551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:28.795553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:28.795637 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:28.796004 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:28.796008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:28.796010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:28.796200 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:28.796356 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:28.796882 (+ 526us) server_negotiation.cc:299] Negotiation successful
0805 23:05:28.797015 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:05:28.798010 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:28.783848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:36707 (local address 127.3.12.65:47227)
0805 23:05:28.784439 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:28.784503 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:28.786136 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:28.786684 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:28.786692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:28.787053 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:28.787534 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:28.787547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:28.788455 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:28.788460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:28.789973 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:28.789988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:28.790532 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:28.790539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:28.790787 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:28.791441 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:28.791468 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:28.793198 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:28.795142 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:28.795148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:28.795164 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:28.795434 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:28.795733 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:28.795737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:28.795739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:28.795904 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:28.796368 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:28.796377 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:28.796632 (+ 255us) client_negotiation.cc:769] Sending connection context
0805 23:05:28.796827 (+ 195us) client_negotiation.cc:240] Negotiation successful
0805 23:05:28.797163 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":165,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:05:28.799324 11604 heartbeater.cc:347] Connected to a master server at 127.3.12.126:36707
I20260805 23:05:28.799592 11604 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:28.800091 11604 heartbeater.cc:511] Master 127.3.12.126:36707 requested a full tablet report, sending...
I20260805 23:05:28.801679 11320 ts_manager.cc:194] Registered new tserver with Master: 2fcdc4eed58f46359665319a5241ba30 (127.3.12.65:34397)
I20260805 23:05:28.802788 11320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:47227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:28.815497 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36707
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:28.928282 11610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:28.928542 11610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:28.928584 11610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:28.932888 11610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:28.932957 11610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:28.933065 11610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:28.938521 11610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:28.939672 11610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:28.940793 11610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:28.950903 11621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:28.950978 11617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:28.950930 11618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:28.951251 11610 server_base.cc:1034] running on GCE node
I20260805 23:05:28.951671 11610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:28.952380 11610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:28.953536 11610 hybrid_clock.cc:648] HybridClock initialized: now 1785971128953505 us; error 48 us; skew 500 ppm
Aug 05 23:05:28 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971128, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:28.956637 11610 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:28.957619 11610 webserver.cc:460] Webserver started at http://127.3.12.66:42177/ using document root <none> and password file <none>
I20260805 23:05:28.958204 11610 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:28.958289 11610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:28.958518 11610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:28.960676 11610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "532a38f2b2fa4917a6f9cc2dfd85e231"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "a835a60f4a0113774cf372acc2f03c2e"
server_key_iv: "8d9037a6d9922b798035ff626d33f6e1"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.961266 11610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "532a38f2b2fa4917a6f9cc2dfd85e231"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "a835a60f4a0113774cf372acc2f03c2e"
server_key_iv: "8d9037a6d9922b798035ff626d33f6e1"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.965117 11610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260805 23:05:28.967576 11632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:28.968840 11610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:28.968984 11610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "532a38f2b2fa4917a6f9cc2dfd85e231"
format_stamp: "Formatted at 2026-08-05 23:05:28 on dist-test-slave-2d19"
server_key: "a835a60f4a0113774cf372acc2f03c2e"
server_key_iv: "8d9037a6d9922b798035ff626d33f6e1"
server_key_version: "encryptionkey@0"
I20260805 23:05:28.969107 11610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:28.988826 11610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:28.991871 11610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:28.992097 11610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:28.992770 11610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:28.993646 11610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:28.993722 11610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:28.993803 11610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:28.993855 11610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:29.002902 11610 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:46317
I20260805 23:05:29.002923 11791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:46317 every 8 connection(s)
I20260805 23:05:29.003875 11610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:29.012811 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 11610
I20260805 23:05:29.012918 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:29.013264 3121 external_mini_cluster.cc:1428] Setting key 821f8c25602b395d66d95886e8da1604
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971128, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:29.018631 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:29.005788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:52455 (local address 127.3.12.126:36707)
0805 23:05:29.005933 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:29.005938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:29.006857 (+ 919us) server_negotiation.cc:407] Connection header received
0805 23:05:29.008247 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:29.008296 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:29.008364 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:29.008445 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:29.009656 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.010182 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.011291 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.011500 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.011581 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:29.014506 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:29.014509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:29.014511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:29.014536 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:29.016337 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.016853 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.016859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.016864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.016966 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.017431 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.017434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.017435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.017622 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:29.017705 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:29.018389 (+ 684us) server_negotiation.cc:299] Negotiation successful
0805 23:05:29.018504 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:05:29.019893 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:29.006204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:36707 (local address 127.3.12.66:52455)
0805 23:05:29.006684 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:29.006730 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:29.008022 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:29.008631 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:29.008639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:29.009013 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:29.009514 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:29.009528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.010292 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.010294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:29.011180 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:29.011190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.011639 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.011642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:29.011786 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:29.012499 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:29.012515 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:29.014326 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:29.016462 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:29.016467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:29.016481 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:29.016747 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:29.017070 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:29.017072 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:29.017074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:29.017309 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:29.017736 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:29.017746 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:29.018115 (+ 369us) client_negotiation.cc:769] Sending connection context
0805 23:05:29.018346 (+ 231us) client_negotiation.cc:240] Negotiation successful
0805 23:05:29.018744 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260805 23:05:29.021180 11792 heartbeater.cc:347] Connected to a master server at 127.3.12.126:36707
I20260805 23:05:29.021426 11792 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:29.021942 11792 heartbeater.cc:511] Master 127.3.12.126:36707 requested a full tablet report, sending...
I20260805 23:05:29.023028 11320 ts_manager.cc:194] Registered new tserver with Master: 532a38f2b2fa4917a6f9cc2dfd85e231 (127.3.12.66:46317)
I20260805 23:05:29.023615 11320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:52455
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:29.045423 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36707
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:29.161952 11797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:29.162218 11797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:29.162261 11797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:29.166209 11797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:29.166265 11797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:29.166345 11797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:29.171468 11797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:29.172576 11797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:29.173661 11797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:29.183704 11807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:29.183712 11804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:29.183763 11797 server_base.cc:1034] running on GCE node
W20260805 23:05:29.183712 11803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:29.184523 11797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:29.185117 11797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:29.186285 11797 hybrid_clock.cc:648] HybridClock initialized: now 1785971129186268 us; error 27 us; skew 500 ppm
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:29.189262 11797 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:29.190181 11797 webserver.cc:460] Webserver started at http://127.3.12.67:39707/ using document root <none> and password file <none>
I20260805 23:05:29.190757 11797 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:29.190820 11797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:29.191057 11797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:29.193208 11797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "afbff945db294ae9b66cd202233760d5"
format_stamp: "Formatted at 2026-08-05 23:05:29 on dist-test-slave-2d19"
server_key: "884548ea08b32000c3eb8ebcb277f6dd"
server_key_iv: "d65170648b246cbd3b63fe51e9d02310"
server_key_version: "encryptionkey@0"
I20260805 23:05:29.193781 11797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "afbff945db294ae9b66cd202233760d5"
format_stamp: "Formatted at 2026-08-05 23:05:29 on dist-test-slave-2d19"
server_key: "884548ea08b32000c3eb8ebcb277f6dd"
server_key_iv: "d65170648b246cbd3b63fe51e9d02310"
server_key_version: "encryptionkey@0"
I20260805 23:05:29.197641 11797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:29.200047 11815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:29.201294 11797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:29.201433 11797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "afbff945db294ae9b66cd202233760d5"
format_stamp: "Formatted at 2026-08-05 23:05:29 on dist-test-slave-2d19"
server_key: "884548ea08b32000c3eb8ebcb277f6dd"
server_key_iv: "d65170648b246cbd3b63fe51e9d02310"
server_key_version: "encryptionkey@0"
I20260805 23:05:29.201560 11797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:29.223381 11797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:29.226716 11797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:29.226944 11797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:29.227584 11797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:29.228564 11797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:29.228634 11797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:29.228708 11797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:29.228756 11797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:29.240830 11797 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:35819
I20260805 23:05:29.240865 11991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:35819 every 8 connection(s)
I20260805 23:05:29.241820 11797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:29.243346 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 11797
I20260805 23:05:29.243505 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:29.243810 3121 external_mini_cluster.cc:1428] Setting key a26f62c022990a2ae9c1a496985ddcf7
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:29.256901 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:29.243994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:51057 (local address 127.3.12.126:36707)
0805 23:05:29.244158 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:29.244162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:29.245131 (+ 969us) server_negotiation.cc:407] Connection header received
0805 23:05:29.246810 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:29.246813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:29.246870 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:29.246965 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:29.248153 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.248764 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.249949 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.250200 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.250315 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:29.253095 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:29.253099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:29.253101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:29.253126 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:29.254517 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.255149 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.255152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.255154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.255242 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.255620 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.255624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.255625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.255818 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:29.255950 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:29.256584 (+ 634us) server_negotiation.cc:299] Negotiation successful
0805 23:05:29.256737 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 23:05:29.257997 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:29.244472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:36707 (local address 127.3.12.67:51057)
0805 23:05:29.244979 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:29.245041 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:29.246597 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:29.247149 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:29.247155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:29.247527 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:29.248010 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:29.248023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.248894 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.248898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:29.249816 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:29.249826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.250362 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.250366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:29.250597 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:29.251210 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:29.251227 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:29.252914 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:29.254730 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:29.254737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:29.254754 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:29.255029 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:29.255347 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:29.255350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:29.255352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:29.255500 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:29.255951 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:29.255959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:29.256226 (+ 267us) client_negotiation.cc:769] Sending connection context
0805 23:05:29.256516 (+ 290us) client_negotiation.cc:240] Negotiation successful
0805 23:05:29.256869 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260805 23:05:29.259152 11992 heartbeater.cc:347] Connected to a master server at 127.3.12.126:36707
I20260805 23:05:29.259361 11992 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:29.259859 11992 heartbeater.cc:511] Master 127.3.12.126:36707 requested a full tablet report, sending...
I20260805 23:05:29.260937 11320 ts_manager.cc:194] Registered new tserver with Master: afbff945db294ae9b66cd202233760d5 (127.3.12.67:35819)
I20260805 23:05:29.261520 11320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:51057
I20260805 23:05:29.269853 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/test-admin.keytab.
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:29.289012 3121 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260805 23:05:29.292407 12003 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:29.290601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.3.12.65:34397)
0805 23:05:29.290989 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:29.291007 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:29.291032 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:05:29.291085 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:29.291090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:29.291293 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:29.291475 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:29.292130 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41626: BlockingRecv error: recv got EOF from 127.0.0.1:41626 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":124,"threads_started":1}
W20260805 23:05:29.292465 3121 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5cc' of type FILE
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:29.293555 3121 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:29.294525 3121 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:05:29.304143 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:29.295436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.3.12.65:34397)
0805 23:05:29.295566 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:29.295569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:29.295622 (+ 53us) server_negotiation.cc:407] Connection header received
0805 23:05:29.295722 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:29.295724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:29.295763 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:29.295830 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:29.296387 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.297166 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.297988 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.298384 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.298496 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:29.300755 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:29.300764 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:29.300778 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:29.300800 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:29.302472 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.302885 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.302890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.302891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.302969 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.303207 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.303209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.303210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.303441 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:29.303595 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:29.303776 (+ 181us) server_negotiation.cc:299] Negotiation successful
0805 23:05:29.303949 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260805 23:05:29.308396 12003 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:29.307379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.3.12.65:34397)
0805 23:05:29.307498 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:29.307501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:29.307656 (+ 155us) server_negotiation.cc:407] Connection header received
0805 23:05:29.307752 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:29.307754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:29.307789 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:29.307853 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:29.308243 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41650: BlockingRecv error: recv got EOF from 127.0.0.1:41650 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260805 23:05:29.308821 3121 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5cc)
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:29.310000 3121 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:29.310981 3121 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:05:29.319936 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:29.311880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.3.12.65:34397)
0805 23:05:29.312004 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:29.312007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:29.312082 (+ 75us) server_negotiation.cc:407] Connection header received
0805 23:05:29.312186 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:29.312188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:29.312223 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:29.312375 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:29.312918 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.313396 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.314303 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.314566 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.314659 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:29.317051 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:29.317054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:29.317056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:29.317078 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:29.318377 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.318801 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.318804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.318805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.318878 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.319143 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.319145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.319146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.319300 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:29.319401 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:29.319725 (+ 324us) server_negotiation.cc:299] Negotiation successful
0805 23:05:29.319821 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260805 23:05:29.322606 12003 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:29.321758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.3.12.65:34397)
0805 23:05:29.321883 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:29.321887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:29.321920 (+ 33us) server_negotiation.cc:407] Connection header received
0805 23:05:29.322039 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:29.322041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:29.322077 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:29.322139 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:29.322506 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41670: BlockingRecv error: recv got EOF from 127.0.0.1:41670 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260805 23:05:29.322741 3121 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785971095475883-3121-0/krb5kdc/krb5cc' of type FILE
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:29.323763 3121 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:29.324704 3121 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:05:29.333444 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:29.325477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.3.12.65:34397)
0805 23:05:29.325599 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:29.325603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:29.325704 (+ 101us) server_negotiation.cc:407] Connection header received
0805 23:05:29.325845 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:29.325848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:29.325885 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:29.326036 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:29.326553 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.327055 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.327856 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:29.328083 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:29.328188 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:29.330404 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:29.330407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:29.330409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:29.330429 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:29.331972 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.332448 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.332451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.332452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.332522 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:29.332751 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:29.332753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:29.332754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:29.332886 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:29.332991 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:29.333161 (+ 170us) server_negotiation.cc:299] Negotiation successful
0805 23:05:29.333242 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:05:29.334385 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 11413
I20260805 23:05:29.350406 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 11610
I20260805 23:05:29.369036 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 11797
I20260805 23:05:29.389869 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 11277
2026-08-05T23:05:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2443 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[12022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[12022](info): set up 2 sockets
Aug 05 23:05:29 dist-test-slave-2d19 krb5kdc[12022](info): commencing operation
krb5kdc: starting...
W20260805 23:05:30.657711 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:30 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.126.keytab.
Entry for principal oryx/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.126.keytab.
I20260805 23:05:30.791824 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:44519
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.126.keytab
--principal=oryx/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:44519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:30.915827 12038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:30.916163 12038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:30.916298 12038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:30.920433 12038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:30.920521 12038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:30.920557 12038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:30.920594 12038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:30.920627 12038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:30.926049 12038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:44519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.126.keytab
--principal=oryx/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:44519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:30.927199 12038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:30.928287 12038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:30.937249 12046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:30.937395 12038 server_base.cc:1034] running on GCE node
W20260805 23:05:30.937252 12048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:30.937330 12050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:30.938208 12038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:30.939301 12038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:30.940554 12038 hybrid_clock.cc:648] HybridClock initialized: now 1785971130940532 us; error 27 us; skew 500 ppm
Aug 05 23:05:30 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971130, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:30.943848 12038 init.cc:377] Logged in from keytab as oryx/127.3.12.126@KRBTEST.COM (short username oryx)
I20260805 23:05:30.944967 12038 webserver.cc:460] Webserver started at http://127.3.12.126:34355/ using document root <none> and password file <none>
I20260805 23:05:30.945566 12038 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:30.945634 12038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:30.945825 12038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:30.948094 12038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "47cc7c4c2a414f83804aa6688a6f6a9e"
format_stamp: "Formatted at 2026-08-05 23:05:30 on dist-test-slave-2d19"
server_key: "f8678b9a81d303abdd673b154e868b92"
server_key_iv: "f44372cff6c6229594d88b720d603208"
server_key_version: "encryptionkey@0"
I20260805 23:05:30.948781 12038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "47cc7c4c2a414f83804aa6688a6f6a9e"
format_stamp: "Formatted at 2026-08-05 23:05:30 on dist-test-slave-2d19"
server_key: "f8678b9a81d303abdd673b154e868b92"
server_key_iv: "f44372cff6c6229594d88b720d603208"
server_key_version: "encryptionkey@0"
I20260805 23:05:30.953066 12038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:30.955765 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:30.957207 12038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:05:30.957362 12038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "47cc7c4c2a414f83804aa6688a6f6a9e"
format_stamp: "Formatted at 2026-08-05 23:05:30 on dist-test-slave-2d19"
server_key: "f8678b9a81d303abdd673b154e868b92"
server_key_iv: "f44372cff6c6229594d88b720d603208"
server_key_version: "encryptionkey@0"
I20260805 23:05:30.957492 12038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:30.970760 12038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:30.974090 12038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:30.974308 12038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:30.984314 12038 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:44519
I20260805 23:05:30.984696 12138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:44519 every 8 connection(s)
I20260805 23:05:30.985858 12038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:30.989579 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 12038
I20260805 23:05:30.989738 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:30.990106 3121 external_mini_cluster.cc:1428] Setting key d24da1b0abf92981f74d113f64aca1b8
I20260805 23:05:30.990510 12139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:30.997036 12139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: Bootstrap starting.
Aug 05 23:05:30 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:30.999531 12139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:31.000325 12139 log.cc:824] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:31.002736 12139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: No bootstrap required, opened a new log
I20260805 23:05:31.005551 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:30.991691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.3.12.126:44519)
0805 23:05:30.992245 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:30.992320 (+ 75us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:30.992372 (+ 52us) server_negotiation.cc:407] Connection header received
0805 23:05:30.993367 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:30.993386 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:30.993729 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:30.994108 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:30.994913 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:30.995913 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:30.996827 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:30.997372 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:30.997546 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:31.000109 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:31.000114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:31.000127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:31.000156 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:31.002284 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:31.002734 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:31.002741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:31.002748 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:31.002897 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:31.003190 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:31.003193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:31.003195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:31.003709 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:31.003915 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:31.004115 (+ 200us) server_negotiation.cc:299] Negotiation successful
0805 23:05:31.004473 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":163,"threads_started":1}
I20260805 23:05:31.007310 12139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } }
I20260805 23:05:31.007562 12139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:31.007644 12139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cc7c4c2a414f83804aa6688a6f6a9e, State: Initialized, Role: FOLLOWER
I20260805 23:05:31.008144 12139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } }
I20260805 23:05:31.008332 12139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:31.008425 12139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:31.008549 12139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:31.009487 12139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } }
I20260805 23:05:31.009847 12139 leader_election.cc:302] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47cc7c4c2a414f83804aa6688a6f6a9e; no voters:
I20260805 23:05:31.010318 12139 leader_election.cc:288] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:31.010394 12144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:31.010686 12144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 1 LEADER]: Becoming Leader. State: Replica: 47cc7c4c2a414f83804aa6688a6f6a9e, State: Running, Role: LEADER
I20260805 23:05:31.011001 12144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } }
I20260805 23:05:31.011701 12139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:31.012749 12144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47cc7c4c2a414f83804aa6688a6f6a9e. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } }
I20260805 23:05:31.012943 12144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:31.013228 12145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } }
I20260805 23:05:31.013330 12158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:31.013334 12145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:31.016469 12158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:31.022876 12158 catalog_manager.cc:1347] Generated new cluster ID: 7281e640834947a28ebb27d1e58020cd
I20260805 23:05:31.022964 12158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:31.036087 12158 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:31.037388 12158 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:31.050174 12158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: Generated new TSK 0
I20260805 23:05:31.051033 12158 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab.
Entry for principal oryx/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab.
I20260805 23:05:31.081561 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:44519
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab
--principal=oryx/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:31.196058 12173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:31.196394 12173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:31.196470 12173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:31.201243 12173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:31.201321 12173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:31.201407 12173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:31.206626 12173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab
--principal=oryx/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:31.207707 12173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:31.208812 12173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:31.218786 12181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:31.218792 12180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:31.218799 12183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:31.218974 12173 server_base.cc:1034] running on GCE node
I20260805 23:05:31.219528 12173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:31.220217 12173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:31.221451 12173 hybrid_clock.cc:648] HybridClock initialized: now 1785971131221429 us; error 28 us; skew 500 ppm
Aug 05 23:05:31 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:31.225200 12173 init.cc:377] Logged in from keytab as oryx/127.3.12.65@KRBTEST.COM (short username oryx)
I20260805 23:05:31.226285 12173 webserver.cc:460] Webserver started at http://127.3.12.65:37387/ using document root <none> and password file <none>
I20260805 23:05:31.226889 12173 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:31.226954 12173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:31.227133 12173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:31.229350 12173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "a616b04439d245d4b24b4b0f3c164363"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "3c8d506ccc56e4968dd3cd7624fdb681"
server_key_iv: "4875d992abaa8e5f47c072b14e57521e"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.230048 12173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "a616b04439d245d4b24b4b0f3c164363"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "3c8d506ccc56e4968dd3cd7624fdb681"
server_key_iv: "4875d992abaa8e5f47c072b14e57521e"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.234571 12173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:05:31.238020 12195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.239665 12173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 23:05:31.239810 12173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "a616b04439d245d4b24b4b0f3c164363"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "3c8d506ccc56e4968dd3cd7624fdb681"
server_key_iv: "4875d992abaa8e5f47c072b14e57521e"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.239938 12173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:31.261181 12173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:31.264451 12173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:31.264662 12173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:31.265287 12173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:31.266155 12173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:31.266211 12173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.266280 12173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:31.266311 12173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.275776 12173 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:37743
I20260805 23:05:31.275797 12366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:37743 every 8 connection(s)
I20260805 23:05:31.276782 12173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:31.279017 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 12173
I20260805 23:05:31.279187 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:31.279546 3121 external_mini_cluster.cc:1428] Setting key 16a77a46e67ccebca7f9e75c0ed79cab
Aug 05 23:05:31 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:31.292972 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:31.278652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:58475 (local address 127.3.12.126:44519)
0805 23:05:31.278824 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:31.278827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:31.279886 (+ 1059us) server_negotiation.cc:407] Connection header received
0805 23:05:31.281628 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:31.281631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:31.281683 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:31.281762 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:31.283015 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.283552 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.285028 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.285496 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.285652 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:31.288730 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:31.288736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:31.288742 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:31.288772 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:31.290594 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:31.291193 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:31.291199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:31.291202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:31.291309 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:31.291675 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:31.291678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:31.291679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:31.291871 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:31.292035 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:31.292728 (+ 693us) server_negotiation.cc:299] Negotiation successful
0805 23:05:31.292845 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260805 23:05:31.293970 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:31.279141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.65:58475)
0805 23:05:31.279726 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:31.279773 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:31.281419 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:31.281935 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:31.281942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:31.282323 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:31.282830 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:31.282854 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.283700 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.283706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:31.284890 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:31.284902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.285668 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.285672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:31.285815 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:31.286625 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:31.286653 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:31.288551 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:31.290722 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:31.290728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:31.290741 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:31.291051 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:31.291410 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:31.291413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:31.291414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:31.291548 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:31.292161 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:31.292171 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:31.292479 (+ 308us) client_negotiation.cc:769] Sending connection context
0805 23:05:31.292648 (+ 169us) client_negotiation.cc:240] Negotiation successful
0805 23:05:31.292920 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.66@KRBTEST.COM" created.
I20260805 23:05:31.295477 12369 heartbeater.cc:347] Connected to a master server at 127.3.12.126:44519
I20260805 23:05:31.295750 12369 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:31.296326 12369 heartbeater.cc:511] Master 127.3.12.126:44519 requested a full tablet report, sending...
I20260805 23:05:31.298084 12085 ts_manager.cc:194] Registered new tserver with Master: a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743)
I20260805 23:05:31.299230 12085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:58475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab.
Entry for principal oryx/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab.
I20260805 23:05:31.311935 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:44519
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab
--principal=oryx/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:31.428076 12375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:31.428434 12375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:31.428526 12375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:31.432552 12375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:31.432631 12375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:31.432717 12375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:31.437979 12375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab
--principal=oryx/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:31.439076 12375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:31.440158 12375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:31.450367 12381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:31.450359 12380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:31.450359 12383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:31.450711 12375 server_base.cc:1034] running on GCE node
I20260805 23:05:31.451130 12375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:31.451797 12375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:31.452986 12375 hybrid_clock.cc:648] HybridClock initialized: now 1785971131452971 us; error 37 us; skew 500 ppm
Aug 05 23:05:31 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:31.455865 12375 init.cc:377] Logged in from keytab as oryx/127.3.12.66@KRBTEST.COM (short username oryx)
I20260805 23:05:31.456779 12375 webserver.cc:460] Webserver started at http://127.3.12.66:38475/ using document root <none> and password file <none>
I20260805 23:05:31.457317 12375 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:31.457379 12375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:31.457543 12375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:31.459975 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "b40a7bc2fdef4f0f808206ad3eeec5be"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "cd4bfc14a08c4567847c07c3dad56321"
server_key_iv: "d65c98bbd7fdee90d3c413337c1eda40"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.460688 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "b40a7bc2fdef4f0f808206ad3eeec5be"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "cd4bfc14a08c4567847c07c3dad56321"
server_key_iv: "d65c98bbd7fdee90d3c413337c1eda40"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.464679 12375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:31.467127 12396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.468451 12375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:31.468587 12375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "b40a7bc2fdef4f0f808206ad3eeec5be"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "cd4bfc14a08c4567847c07c3dad56321"
server_key_iv: "d65c98bbd7fdee90d3c413337c1eda40"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.468703 12375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:31.501361 12375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:31.504911 12375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:31.505141 12375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:31.505769 12375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:31.506611 12375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:31.506682 12375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.506755 12375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:31.506807 12375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.516258 12375 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:40099
I20260805 23:05:31.516302 12559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:40099 every 8 connection(s)
I20260805 23:05:31.517262 12375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:31.519583 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 12375
I20260805 23:05:31.519708 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:31.520063 3121 external_mini_cluster.cc:1428] Setting key e761d63e8aa66f4dae562de9f0ff490b
Aug 05 23:05:31 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:05:31.531631 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:31.519156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:45573 (local address 127.3.12.126:44519)
0805 23:05:31.519300 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:31.519304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:31.520475 (+ 1171us) server_negotiation.cc:407] Connection header received
0805 23:05:31.521997 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:31.522000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:31.522058 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:31.522137 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:31.523333 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.523865 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.525116 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.525348 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.525461 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:31.528034 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:31.528037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:31.528040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:31.528069 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:31.529573 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:31.530088 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:31.530092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:31.530094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:31.530176 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:31.530515 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:31.530520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:31.530522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:31.530711 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:31.530819 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:31.531329 (+ 510us) server_negotiation.cc:299] Negotiation successful
0805 23:05:31.531460 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260805 23:05:31.532588 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:31.519693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:45573)
0805 23:05:31.520322 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:31.520370 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:31.521793 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:31.522310 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:31.522317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:31.522695 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:31.523189 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:31.523203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.524004 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.524007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:31.524982 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:31.524993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.525506 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.525511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:31.525691 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:31.526313 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:31.526330 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:31.527861 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:31.529696 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:31.529702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:31.529718 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:31.529980 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:31.530268 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:31.530272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:31.530275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:31.530419 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:31.530819 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:31.530827 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:31.531091 (+ 264us) client_negotiation.cc:769] Sending connection context
0805 23:05:31.531288 (+ 197us) client_negotiation.cc:240] Negotiation successful
0805 23:05:31.531635 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.67@KRBTEST.COM" created.
I20260805 23:05:31.533821 12560 heartbeater.cc:347] Connected to a master server at 127.3.12.126:44519
I20260805 23:05:31.534097 12560 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:31.534616 12560 heartbeater.cc:511] Master 127.3.12.126:44519 requested a full tablet report, sending...
I20260805 23:05:31.535673 12085 ts_manager.cc:194] Registered new tserver with Master: b40a7bc2fdef4f0f808206ad3eeec5be (127.3.12.66:40099)
I20260805 23:05:31.536288 12085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:45573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab.
Entry for principal oryx/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab.
I20260805 23:05:31.551733 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:44519
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab
--principal=oryx/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:31.667863 12566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:31.668213 12566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:31.668318 12566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:31.672394 12566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:31.672452 12566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:31.672511 12566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:31.677918 12566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab
--principal=oryx/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:44519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:31.679056 12566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:31.680096 12566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:31.690752 12572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:31.690752 12571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:31.690901 12566 server_base.cc:1034] running on GCE node
W20260805 23:05:31.691095 12575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:31.691498 12566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:31.692202 12566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:31.693423 12566 hybrid_clock.cc:648] HybridClock initialized: now 1785971131693382 us; error 55 us; skew 500 ppm
Aug 05 23:05:31 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:31.696434 12566 init.cc:377] Logged in from keytab as oryx/127.3.12.67@KRBTEST.COM (short username oryx)
I20260805 23:05:31.697331 12566 webserver.cc:460] Webserver started at http://127.3.12.67:45447/ using document root <none> and password file <none>
I20260805 23:05:31.697873 12566 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:31.697932 12566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:31.698105 12566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:31.700202 12566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "6f097662686240abbf18cd043df2a7f5"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "4243acd7b536c459d67b369409836cce"
server_key_iv: "61296f7c42d382c687dd42c11230fc55"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.700814 12566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "6f097662686240abbf18cd043df2a7f5"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "4243acd7b536c459d67b369409836cce"
server_key_iv: "61296f7c42d382c687dd42c11230fc55"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.704560 12566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:31.706888 12591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.708300 12566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:31.708442 12566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "6f097662686240abbf18cd043df2a7f5"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "4243acd7b536c459d67b369409836cce"
server_key_iv: "61296f7c42d382c687dd42c11230fc55"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.708562 12566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:31.724411 12566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:31.727821 12566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:31.728037 12566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:31.728775 12566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:31.729642 12566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:31.729715 12566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.729791 12566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:31.729835 12566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.739006 12566 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:46457
I20260805 23:05:31.739077 12745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:46457 every 8 connection(s)
I20260805 23:05:31.739966 12566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:31.749475 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 12566
I20260805 23:05:31.749641 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:31.749982 3121 external_mini_cluster.cc:1428] Setting key 686986fd9f1cee73fc511cbe23a946e4
Aug 05 23:05:31 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:31.754388 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:31.741872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:48151 (local address 127.3.12.126:44519)
0805 23:05:31.742013 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:31.742016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:31.742983 (+ 967us) server_negotiation.cc:407] Connection header received
0805 23:05:31.744431 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:31.744435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:31.744495 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:31.744614 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:31.745814 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.746414 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.747599 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.747827 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.747944 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:31.750717 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:31.750722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:31.750725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:31.750756 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:31.752534 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:31.753031 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:31.753034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:31.753035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:31.753126 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:31.753435 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:31.753437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:31.753438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:31.753616 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:31.753692 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:31.754168 (+ 476us) server_negotiation.cc:299] Negotiation successful
0805 23:05:31.754269 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:05:31.755291 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:31.742349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:48151)
0805 23:05:31.742839 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:31.742884 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:31.744165 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:31.744794 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:31.744800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:31.745185 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:31.745666 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:31.745680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.746548 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.746552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:31.747448 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:31.747459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:31.747991 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:31.747995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:31.748173 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:31.748797 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:31.748814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:31.750513 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:31.752650 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:31.752655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:31.752669 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:31.752938 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:31.753213 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:31.753215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:31.753216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:31.753353 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:31.753706 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:31.753711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:31.753949 (+ 238us) client_negotiation.cc:769] Sending connection context
0805 23:05:31.754120 (+ 171us) client_negotiation.cc:240] Negotiation successful
0805 23:05:31.754385 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260805 23:05:31.756623 12746 heartbeater.cc:347] Connected to a master server at 127.3.12.126:44519
I20260805 23:05:31.756872 12746 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:31.757390 12746 heartbeater.cc:511] Master 127.3.12.126:44519 requested a full tablet report, sending...
I20260805 23:05:31.758577 12085 ts_manager.cc:194] Registered new tserver with Master: 6f097662686240abbf18cd043df2a7f5 (127.3.12.67:46457)
I20260805 23:05:31.759157 12085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:48151
I20260805 23:05:31.764235 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.125.keytab.
Entry for principal oryx/127.3.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.125.keytab.
I20260805 23:05:31.795940 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.125:34147
--webserver_interface=127.3.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.125.keytab
--principal=oryx/127.3.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:44519,127.3.12.125:34147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:31.913777 12752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:31.914119 12752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:31.914203 12752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:31.918377 12752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:31.918447 12752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:31.918468 12752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:31.918529 12752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:31.918558 12752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:31.924139 12752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:44519,127.3.12.125:34147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.125.keytab
--principal=oryx/127.3.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.125:34147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
W20260805 23:05:31.924423 12752 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.12.126:44519,127.3.12.125:34147'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260805 23:05:31.925338 12752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:31.926322 12752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:31.935258 12759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:31.935258 12762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:31.935356 12758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:31.935755 12752 server_base.cc:1034] running on GCE node
I20260805 23:05:31.936386 12752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:31.937476 12752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:31.938645 12752 hybrid_clock.cc:648] HybridClock initialized: now 1785971131938630 us; error 27 us; skew 500 ppm
Aug 05 23:05:31 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:31.941455 12752 init.cc:377] Logged in from keytab as oryx/127.3.12.125@KRBTEST.COM (short username oryx)
I20260805 23:05:31.942407 12752 webserver.cc:460] Webserver started at http://127.3.12.125:36465/ using document root <none> and password file <none>
I20260805 23:05:31.943073 12752 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:31.943137 12752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:31.943399 12752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:31.945606 12752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data/instance:
uuid: "eee475496b804e56ac72865c49d6fad0"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "91902acff66e152cb48aea2924d6e3e4"
server_key_iv: "6361f79e3a5d746e348695e6923a3ba8"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.946213 12752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal/instance:
uuid: "eee475496b804e56ac72865c49d6fad0"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "91902acff66e152cb48aea2924d6e3e4"
server_key_iv: "6361f79e3a5d746e348695e6923a3ba8"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.950034 12752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:05:31.952538 12772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:31.953835 12752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:31.953990 12752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
uuid: "eee475496b804e56ac72865c49d6fad0"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "91902acff66e152cb48aea2924d6e3e4"
server_key_iv: "6361f79e3a5d746e348695e6923a3ba8"
server_key_version: "encryptionkey@0"
I20260805 23:05:31.954121 12752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:31.981554 12752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:31.985183 12752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:31.985400 12752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:31.994696 12752 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.125:34147
I20260805 23:05:31.994990 12853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.125:34147 every 8 connection(s)
I20260805 23:05:31.995992 12752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
I20260805 23:05:31.998900 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:32.002772 12855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:05:32.003355 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 12752
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.125@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.12.125@KRBTEST.COM
I20260805 23:05:32.019375 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.004861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.3.12.126:44519)
0805 23:05:32.005009 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.005014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.005596 (+ 582us) server_negotiation.cc:407] Connection header received
0805 23:05:32.006380 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.006384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.006444 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.006537 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.007373 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.007901 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.008976 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.009194 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.009304 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.012029 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.012035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.012038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.012097 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.016947 (+ 4850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.017614 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.017618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.017619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.017717 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.018151 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.018154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.018156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.018461 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.018576 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.019046 (+ 470us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.019195 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:05:32.020366 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.004935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.3.12.125:34147)
0805 23:05:32.005343 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.005361 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.005402 (+ 41us) server_negotiation.cc:407] Connection header received
0805 23:05:32.006158 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.006176 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.006373 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.006580 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.007423 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.008315 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.009133 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.009496 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.009621 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.012658 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.012668 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.012707 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.012738 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.017398 (+ 4660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.017942 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.017947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.017949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.018031 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.018354 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.018357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.018359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.018779 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.018994 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.019261 (+ 267us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.019732 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260805 23:05:32.020363 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.004935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60136)
0805 23:05:32.005444 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.005480 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.006170 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.006596 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.006602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.006964 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.007224 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.007238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.008044 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.008048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.008841 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.008850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.009347 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.009351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.009524 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.010130 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.010147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.011835 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.017084 (+ 5249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.017095 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.017107 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.017514 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.017819 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.017821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.017823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.018031 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.018593 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.018602 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.018816 (+ 214us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.019000 (+ 184us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.019344 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"spinlock_wait_cycles":1792,"thread_start_us":168,"threads_started":1}
I20260805 23:05:32.021929 12855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971131, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.125@KRBTEST.COM for oryx/127.3.12.125@KRBTEST.COM
I20260805 23:05:32.030527 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.022435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.0.0.1:46104)
0805 23:05:32.022529 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.022541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.022639 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.022997 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.023000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.023135 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.023226 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.023231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.023920 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.023923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.024516 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.024524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.025611 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.025614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.025687 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.026122 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.026133 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.027368 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.029034 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.029037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.029039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.029234 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.029510 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.029513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.029515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.029593 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.029951 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.029954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.030044 (+ 90us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.030154 (+ 110us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.030298 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260805 23:05:32.030527 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.022735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.3.12.125:34147)
0805 23:05:32.022835 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.022837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.022845 (+ 8us) server_negotiation.cc:407] Connection header received
0805 23:05:32.022869 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.022871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.022901 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.022957 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.023323 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.023829 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.024634 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.025501 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.025596 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.027512 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.027515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.027517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.027536 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.028916 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.029330 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.029335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.029337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.029413 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.029682 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.029687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.029688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.029856 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.029961 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.030209 (+ 248us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.030352 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":45}
I20260805 23:05:32.033553 12855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Bootstrap starting.
I20260805 23:05:32.035959 12855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:32.036818 12855 log.cc:824] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:32.038789 12855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: No bootstrap required, opened a new log
I20260805 23:05:32.042289 12855 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } }
I20260805 23:05:32.042577 12855 raft_consensus.cc:374] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:32.042673 12855 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee475496b804e56ac72865c49d6fad0, State: Initialized, Role: FOLLOWER
I20260805 23:05:32.043291 12855 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } }
I20260805 23:05:32.043900 12864 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } } }
I20260805 23:05:32.044010 12864 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:05:32.044826 12855 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: configured and running, proceeding with master startup.
W20260805 23:05:32.049676 12884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:05:32.049768 12884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:05:32.051487 12752 master_runner.cc:428] Detected that this master eee475496b804e56ac72865c49d6fad0 is joining an existing cluster
I20260805 23:05:32.051560 12752 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.12.125:34147
I20260805 23:05:32.062058 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.053693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.0.0.1:46120)
0805 23:05:32.054200 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.054210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.054315 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.054609 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.054611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.054841 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.054957 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.054966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.055666 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.055671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.056557 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.056567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.057177 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.057180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.057248 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.057818 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.057831 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.058382 (+ 551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.060336 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.060339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.060342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.060583 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.060913 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.060916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.060918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.061008 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.061489 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.061491 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.061555 (+ 64us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.061664 (+ 109us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.061825 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":113,"threads_started":1}
I20260805 23:05:32.062376 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.054016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.3.12.125:34147)
0805 23:05:32.054115 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.054121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.054282 (+ 161us) server_negotiation.cc:407] Connection header received
0805 23:05:32.054386 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.054392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.054432 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.054503 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.055063 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.055551 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.056676 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.057007 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.057060 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.058513 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.058516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.058519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.058542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.060203 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.060700 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.060706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.060709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.060799 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.061130 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.061135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.061137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.061360 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.061429 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.061723 (+ 294us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.061951 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":32}
I20260805 23:05:32.063416 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.053616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60144)
0805 23:05:32.053920 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.053936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.054035 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.054428 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.054430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.054720 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.054873 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.054880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.055968 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.055971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.056870 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.056877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.057605 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.057609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.057711 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.058230 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.058246 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.059156 (+ 910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.061319 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.061322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.061325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.061569 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.062018 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.062021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.062023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.062138 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.062916 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.062918 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.062991 (+ 73us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.063101 (+ 110us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.063242 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":109,"threads_started":1}
I20260805 23:05:32.063431 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.053981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.3.12.126:44519)
0805 23:05:32.054121 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.054126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.054141 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:32.054186 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.054189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.054237 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.054327 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.055020 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.055849 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.056985 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.057344 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.057476 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.059355 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.059359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.059361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.059391 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.061213 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.061815 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.061819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.061821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.061910 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.062531 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.062536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.062539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.062774 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.062851 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.063164 (+ 313us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.063283 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971130, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.126@KRBTEST.COM for oryx/127.3.12.125@KRBTEST.COM
I20260805 23:05:32.077973 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.068187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.3.12.125:34147)
0805 23:05:32.068356 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.068360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.068675 (+ 315us) server_negotiation.cc:407] Connection header received
0805 23:05:32.068892 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.068895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.068950 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.069038 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.069695 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.070186 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.071071 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.071290 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.071406 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.074168 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.074173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.074175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.074201 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.075875 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.076595 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.076601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.076606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.076702 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.077003 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.077006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.077008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.077193 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.077306 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.077675 (+ 369us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.077800 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260805 23:05:32.077977 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.068152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.0.0.1:46136)
0805 23:05:32.068545 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.068593 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.068758 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.069082 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.069089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.069461 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.069573 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.069580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.070300 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.070303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.070945 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.070954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.071410 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.071414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.071575 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.072174 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.072187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.074024 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.076001 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.076014 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.076024 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.076441 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.076803 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.076807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.076809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.076899 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.077307 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.077318 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.077448 (+ 130us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.077596 (+ 148us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.077804 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":127,"threads_started":1}
I20260805 23:05:32.079744 12085 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.12.125:34147: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eee475496b804e56ac72865c49d6fad0"
member_type: NON_VOTER
last_known_addr {
host: "127.3.12.125"
port: 34147
}
attrs {
promote: true
}
}
dest_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e"
cas_config_opid_index: -1
I20260805 23:05:32.080510 12085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } }
I20260805 23:05:32.082288 12900 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER eee475496b804e56ac72865c49d6fad0 (127.3.12.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } }
I20260805 23:05:32.083024 12815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:32.083914 12900 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.12.125:34147
I20260805 23:05:32.084345 12900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } }
I20260805 23:05:32.084408 12752 master.cc:537] Master@127.3.12.125:34147 shutting down...
I20260805 23:05:32.084457 12900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:32.084875 12815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Refusing update from remote peer 47cc7c4c2a414f83804aa6688a6f6a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260805 23:05:32.085675 12901 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.3.12.124@KRBTEST.COM; defaulting to no policy
W20260805 23:05:32.088299 12061 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260805 23:05:32.089288 12061 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e -> Peer eee475496b804e56ac72865c49d6fad0 (127.3.12.125:34147): Couldn't send request to peer eee475496b804e56ac72865c49d6fad0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260805 23:05:32.089502 12752 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:32.089810 12752 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:32.089928 12752 tablet_replica.cc:330] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.124@KRBTEST.COM" created.
I20260805 23:05:32.105544 12752 master.cc:552] Master@127.3.12.125:34147 shutdown complete.
I20260805 23:05:32.105618 12752 master_runner.cc:305] Clearing existing system tablet
I20260805 23:05:32.105906 12752 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:32.105962 12752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.124.keytab.
Entry for principal oryx/127.3.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.124.keytab.
I20260805 23:05:32.107390 12752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260805 23:05:32.107438 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.124:42281
--webserver_interface=127.3.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.124.keytab
--principal=oryx/127.3.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:44519,127.3.12.125:34147,127.3.12.124:42281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:05:32.108385 12908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.108600 12752 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:32.108748 12752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:32.108784 12752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
uuid: "eee475496b804e56ac72865c49d6fad0"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "91902acff66e152cb48aea2924d6e3e4"
server_key_iv: "6361f79e3a5d746e348695e6923a3ba8"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.111302 12752 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:32.113725 12752 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260805 23:05:32.113834 12752 log.cc:1196] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260805 23:05:32.114140 12752 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Deleting consensus metadata
I20260805 23:05:32.114344 12752 master_runner.cc:315] Copying system tablet from 127.3.12.126:44519
I20260805 23:05:32.115505 12752 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: tablet copy: Beginning tablet copy session from remote peer at address 127.3.12.126:44519
I20260805 23:05:32.123657 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.116133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60148)
0805 23:05:32.116481 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.116493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.116635 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.116922 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.116925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.117145 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.117249 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.117255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.118033 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.118037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.118631 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.118640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.119120 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.119124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.119230 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.119799 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.119811 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.120191 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.122043 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.122056 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.122058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.122266 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.122552 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.122557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.122561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.122653 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.123073 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.123076 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.123181 (+ 105us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.123290 (+ 109us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.123441 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":87,"threads_started":1}
I20260805 23:05:32.123647 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.116186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.3.12.126:44519)
0805 23:05:32.116351 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.116356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.116582 (+ 226us) server_negotiation.cc:407] Connection header received
0805 23:05:32.116760 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.116763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.116819 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.116905 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.117376 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.117915 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.118757 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.118996 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.119112 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.120401 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.120405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.120408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.120435 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.121927 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.122372 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.122375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.122377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.122459 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.122746 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.122751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.122752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.122964 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.123075 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.123360 (+ 285us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.123476 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260805 23:05:32.124612 12118 tablet_copy_service.cc:140] P 47cc7c4c2a414f83804aa6688a6f6a9e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eee475496b804e56ac72865c49d6fad0 ({username='oryx', principal='oryx/127.3.12.125@KRBTEST.COM'} at 127.0.0.1:60148)
I20260805 23:05:32.124796 12118 tablet_copy_service.cc:161] P 47cc7c4c2a414f83804aa6688a6f6a9e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eee475496b804e56ac72865c49d6fad0 at {username='oryx', principal='oryx/127.3.12.125@KRBTEST.COM'} at 127.0.0.1:60148: session id = eee475496b804e56ac72865c49d6fad0-00000000000000000000000000000000
I20260805 23:05:32.126251 12118 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 23:05:32.127776 12752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260805 23:05:32.129751 12752 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: tablet copy: Starting download of 0 data blocks...
I20260805 23:05:32.129971 12752 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: tablet copy: Starting download of 1 WAL segments...
I20260805 23:05:32.131644 12752 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 23:05:32.133334 12118 tablet_copy_service.cc:342] P 47cc7c4c2a414f83804aa6688a6f6a9e: Request end of tablet copy session eee475496b804e56ac72865c49d6fad0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.12.125@KRBTEST.COM'} at 127.0.0.1:60148
I20260805 23:05:32.133458 12118 tablet_copy_service.cc:433] P 47cc7c4c2a414f83804aa6688a6f6a9e: ending tablet copy session eee475496b804e56ac72865c49d6fad0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eee475496b804e56ac72865c49d6fad0
I20260805 23:05:32.140957 12752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:32.142602 12925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:32.143126 12929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:32.143332 12752 server_base.cc:1034] running on GCE node
W20260805 23:05:32.143836 12926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:32.144083 12752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:32.144452 12752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:32.145591 12752 hybrid_clock.cc:648] HybridClock initialized: now 1785971132145579 us; error 32 us; skew 500 ppm
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:32.147037 12752 init.cc:377] Logged in from keytab as oryx/127.3.12.125@KRBTEST.COM (short username oryx)
I20260805 23:05:32.147526 12752 webserver.cc:460] Webserver started at http://127.3.12.125:41523/ using document root <none> and password file <none>
I20260805 23:05:32.147720 12752 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:32.147775 12752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:32.148998 12752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:32.149962 12937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.150202 12752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:32.150281 12752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
uuid: "eee475496b804e56ac72865c49d6fad0"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "91902acff66e152cb48aea2924d6e3e4"
server_key_iv: "6361f79e3a5d746e348695e6923a3ba8"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.150365 12752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:32.164214 12752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:32.166328 12752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:32.171589 12752 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.125:34147
I20260805 23:05:32.172417 13007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.125:34147 every 8 connection(s)
I20260805 23:05:32.172466 12752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
I20260805 23:05:32.174351 13010 sys_catalog.cc:263] Verifying existing consensus state
I20260805 23:05:32.175169 13010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Bootstrap starting.
I20260805 23:05:32.183482 13010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:05:32.183845 13010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Bootstrap complete.
I20260805 23:05:32.184420 13010 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } }
I20260805 23:05:32.184564 13010 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eee475496b804e56ac72865c49d6fad0, State: Initialized, Role: LEARNER
I20260805 23:05:32.184681 13010 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } }
I20260805 23:05:32.185042 13015 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } }
I20260805 23:05:32.185201 13010 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:32.185220 13015 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: This master's current role is: LEARNER
I20260805 23:05:32.188683 13030 catalog_manager.cc:1259] Loaded cluster ID: 7281e640834947a28ebb27d1e58020cd
I20260805 23:05:32.188766 13030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: loading cluster ID for follower catalog manager: success
I20260805 23:05:32.190254 13030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: acquiring CA information for follower catalog manager: success
I20260805 23:05:32.191355 13030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260805 23:05:32.268152 12913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:32.268534 12913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:32.268596 12913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:32.272666 12913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:32.272732 12913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:32.272753 12913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:32.272769 12913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:32.272785 12913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:32.278553 12913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:44519,127.3.12.125:34147,127.3.12.124:42281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.124.keytab
--principal=oryx/127.3.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.124:42281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:32.279623 12913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:32.280705 12913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:32.289695 13037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:32.289688 13036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:32.289716 13041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:32.289916 12913 server_base.cc:1034] running on GCE node
I20260805 23:05:32.290501 12913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:32.291659 12913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:32.292826 12913 hybrid_clock.cc:648] HybridClock initialized: now 1785971132292819 us; error 38 us; skew 500 ppm
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:32.295720 12913 init.cc:377] Logged in from keytab as oryx/127.3.12.124@KRBTEST.COM (short username oryx)
I20260805 23:05:32.296725 12913 webserver.cc:460] Webserver started at http://127.3.12.124:34807/ using document root <none> and password file <none>
I20260805 23:05:32.297335 12913 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:32.297434 12913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:32.297673 12913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:32.299752 12913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data/instance:
uuid: "504c0d936fa440e8bab92a2cee3d7c40"
format_stamp: "Formatted at 2026-08-05 23:05:32 on dist-test-slave-2d19"
server_key: "b9e8ae7fb6856b083a238f76f7e5b355"
server_key_iv: "9764a71498b7e8dc17904eab97e51339"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.300414 12913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal/instance:
uuid: "504c0d936fa440e8bab92a2cee3d7c40"
format_stamp: "Formatted at 2026-08-05 23:05:32 on dist-test-slave-2d19"
server_key: "b9e8ae7fb6856b083a238f76f7e5b355"
server_key_iv: "9764a71498b7e8dc17904eab97e51339"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.301762 12369 heartbeater.cc:503] Master 127.3.12.126:44519 was elected leader, sending a full tablet report...
I20260805 23:05:32.304734 12913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:32.307469 13055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.308790 12913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:32.308892 12913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
uuid: "504c0d936fa440e8bab92a2cee3d7c40"
format_stamp: "Formatted at 2026-08-05 23:05:32 on dist-test-slave-2d19"
server_key: "b9e8ae7fb6856b083a238f76f7e5b355"
server_key_iv: "9764a71498b7e8dc17904eab97e51339"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.308977 12913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:32.325835 12913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:32.328995 12913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:32.329162 12913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:32.336691 12913 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.124:42281
I20260805 23:05:32.336994 13124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.124:42281 every 8 connection(s)
I20260805 23:05:32.337854 12913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
I20260805 23:05:32.341728 13126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:32.345870 13126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:05:32.346623 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 12913
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.124@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.12.124@KRBTEST.COM
I20260805 23:05:32.360706 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.348100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.3.12.126:44519)
0805 23:05:32.348234 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.348237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.351025 (+ 2788us) server_negotiation.cc:407] Connection header received
0805 23:05:32.351087 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.351089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.351141 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.351227 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.352185 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.352791 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.353837 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.354064 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.354181 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.356811 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.356815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.356818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.356847 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.358507 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.359063 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.359068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.359070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.359164 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.359555 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.359561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.359563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.359767 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.359892 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.360374 (+ 482us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.360508 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:05:32.361639 13129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.348291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60154)
0805 23:05:32.350811 (+ 2520us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.350870 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.351004 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.351316 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.351321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.351794 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.352015 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.352028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.352945 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.352948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.353701 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.353710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.354227 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.354231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.354345 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.354903 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.354918 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.356643 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.358636 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.358648 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.358659 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.358947 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.359271 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.359275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.359277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.359439 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.359890 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.359898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.360107 (+ 209us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.360332 (+ 225us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.360690 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2301,"thread_start_us":188,"threads_started":1}
I20260805 23:05:32.361639 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.348286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.3.12.124:42281)
0805 23:05:32.348771 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.348785 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.348820 (+ 35us) server_negotiation.cc:407] Connection header received
0805 23:05:32.349470 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.349488 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.349873 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.350180 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.351316 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.352680 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.353517 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.353915 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.354082 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.357057 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.357064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.357074 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.357095 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.358829 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.359277 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.359282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.359294 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.359435 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.359725 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.359729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.359731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.360096 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.360342 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.360524 (+ 182us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.360787 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":349,"spinlock_wait_cycles":52992,"thread_start_us":188,"threads_started":1}
I20260805 23:05:32.363050 13126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.124@KRBTEST.COM for oryx/127.3.12.125@KRBTEST.COM
I20260805 23:05:32.372548 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.363571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.0.0.1:46144)
0805 23:05:32.364219 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.364255 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.364380 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.364641 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.364645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.364845 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.364933 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.364938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.365865 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.365870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.366442 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.366450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.366938 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.366942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.367040 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.368008 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.368019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.369437 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.371045 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.371048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.371049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.371260 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.371537 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.371539 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.371540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.371618 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.371980 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.371983 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.372065 (+ 82us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.372152 (+ 87us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.372326 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":209,"threads_started":1}
I20260805 23:05:32.372581 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.363688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.3.12.125:34147)
0805 23:05:32.364060 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.364074 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.364344 (+ 270us) server_negotiation.cc:407] Connection header received
0805 23:05:32.364466 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.364469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.364522 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.364629 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.365060 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.365751 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.366577 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.366814 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.366936 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.369597 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.369601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.369602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.369627 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.370938 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.371368 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.371372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.371373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.371453 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.371695 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.371698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.371699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.371871 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.371984 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.372213 (+ 229us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.372419 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
I20260805 23:05:32.373416 13126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.124@KRBTEST.COM for oryx/127.3.12.124@KRBTEST.COM
I20260805 23:05:32.381242 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.373928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42281 (local address 127.0.0.1:36400)
0805 23:05:32.374049 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.374065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.374140 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.374369 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.374372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.374537 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.374624 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.374628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.375339 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.375344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.375864 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.375872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.376376 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.376380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.376497 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.376937 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.376947 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.378137 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.379682 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.379687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.379689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.379875 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.380181 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.380186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.380188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.380320 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.380707 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.380710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.380791 (+ 81us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.380903 (+ 112us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.381049 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":18}
I20260805 23:05:32.381242 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.374087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.3.12.124:42281)
0805 23:05:32.374193 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.374198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.374211 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:05:32.374237 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.374239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.374278 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.374358 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.374732 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.375234 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.375978 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.376226 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.376379 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.378260 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.378263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.378266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.378288 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.379575 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.379986 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.379992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.379994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.380093 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.380425 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.380428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.380430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.380608 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.380718 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.380960 (+ 242us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.381108 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":31}
I20260805 23:05:32.384605 13126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Bootstrap starting.
I20260805 23:05:32.387602 13126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:32.388473 13126 log.cc:824] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:32.390492 13126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: No bootstrap required, opened a new log
I20260805 23:05:32.393594 13126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } }
I20260805 23:05:32.393810 13126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:32.393844 13126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504c0d936fa440e8bab92a2cee3d7c40, State: Initialized, Role: FOLLOWER
I20260805 23:05:32.394308 13126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } }
I20260805 23:05:32.394770 13134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } } }
I20260805 23:05:32.394927 13134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:05:32.395509 13126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: configured and running, proceeding with master startup.
W20260805 23:05:32.400415 13152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:05:32.400508 13152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:05:32.404341 12913 master_runner.cc:428] Detected that this master 504c0d936fa440e8bab92a2cee3d7c40 is joining an existing cluster
I20260805 23:05:32.404397 12913 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.12.124:42281
I20260805 23:05:32.415489 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 12173
I20260805 23:05:32.434507 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.407032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.3.12.124:42281)
0805 23:05:32.407138 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.407142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.407588 (+ 446us) server_negotiation.cc:407] Connection header received
0805 23:05:32.407682 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.407684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.407722 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.407794 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.408927 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.409626 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.410716 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.411052 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.411127 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.412967 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.412970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.412973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.413003 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.414689 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.415131 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.415134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.415136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.415218 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.415498 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.415502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.415504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.415719 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.415809 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.434084 (+ 18275us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.434264 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260805 23:05:32.435184 13165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.406979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42281 (local address 127.0.0.1:36410)
0805 23:05:32.407491 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.407502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.407612 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.407890 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.407896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.408282 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.408819 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.408828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.409730 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.409733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.410605 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.410616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.411267 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.411271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.411763 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.412382 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.412394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.412833 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.414802 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.414805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.414807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.415012 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.415315 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.415317 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.415319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.415401 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.433661 (+ 18260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.433669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.433856 (+ 187us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.434768 (+ 912us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.434957 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":105,"spinlock_wait_cycles":1408,"thread_start_us":103,"threads_started":1}
I20260805 23:05:32.439591 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.406552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.3.12.126:44519)
0805 23:05:32.406748 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.406753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.407354 (+ 601us) server_negotiation.cc:407] Connection header received
0805 23:05:32.407815 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.407818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.407867 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.407921 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.408657 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.409503 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.410712 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.411052 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.411129 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.436117 (+ 24988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.436122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.436124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.436167 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.437767 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.438268 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.438272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.438274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.438368 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.438670 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.438673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.438675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.438881 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.438995 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.439281 (+ 286us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.439417 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260805 23:05:32.439649 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.406681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60164)
0805 23:05:32.407274 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.407290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.407703 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.408088 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.408091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.408319 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.408504 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.408512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.409635 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.409637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.410567 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.410578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.411267 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.411271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.411763 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.435506 (+ 23743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.435522 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.435924 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.437921 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.437927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.437930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.438155 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.438477 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.438480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.438482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.438573 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.438984 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.438987 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.439105 (+ 118us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.439218 (+ 113us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.439466 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":412,"spinlock_wait_cycles":4352,"thread_start_us":101,"threads_started":1}
I20260805 23:05:32.444160 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:37743
--local_ip_for_outbound_sockets=127.3.12.65
--tserver_master_addrs=127.3.12.126:44519
--webserver_port=37387
--webserver_interface=127.3.12.65
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab
--principal=oryx/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147,127.3.12.124:42281 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:05:32.445885 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.406600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.3.12.125:34147)
0805 23:05:32.406856 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.406863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.407359 (+ 496us) server_negotiation.cc:407] Connection header received
0805 23:05:32.407468 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.407471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.407525 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.407619 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.408380 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.409227 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.410454 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.410829 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.410910 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.441620 (+ 30710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.441625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.441628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.441662 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.443584 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.444103 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.444107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.444109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.444204 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.444903 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.444910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.444913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.445141 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.445215 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.445576 (+ 361us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.445720 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260805 23:05:32.446789 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.406542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.0.0.1:46146)
0805 23:05:32.407244 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.407259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.407377 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.407749 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.407753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.408059 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.408194 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.408202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.409377 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.409380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.410313 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.410324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.411088 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.411092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.411191 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.411685 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.411700 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.441395 (+ 29695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.443723 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.443727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.443729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.443985 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.444696 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.444699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.444701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.444793 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.445291 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.445294 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.445390 (+ 96us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.446401 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.446577 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":107,"threads_started":1}
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971130, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.126@KRBTEST.COM for oryx/127.3.12.124@KRBTEST.COM
I20260805 23:05:32.455042 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.444476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.3.12.124:42281)
0805 23:05:32.445088 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.445095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.445110 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:32.445184 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.445190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.445255 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.445328 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.446018 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.446555 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.447790 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.448157 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.448308 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.451127 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.451132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.451135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.451166 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.453041 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.453605 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.453608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.453610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.453700 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.454057 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.454060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.454062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.454289 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.454389 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.454753 (+ 364us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.454876 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520}
I20260805 23:05:32.455149 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.444621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42281 (local address 127.0.0.1:36414)
0805 23:05:32.444820 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.444837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.444922 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.445479 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.445485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.445721 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.445863 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.445873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.446681 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.446685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.447626 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.447641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.448426 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.448430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.448559 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.449119 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.449134 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.450970 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.453175 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.453180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.453184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.453486 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.453814 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.453820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.453825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.453948 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.454406 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.454413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.454552 (+ 139us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.454689 (+ 137us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.454887 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260805 23:05:32.457000 12085 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.12.124:42281: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40"
member_type: NON_VOTER
last_known_addr {
host: "127.3.12.124"
port: 42281
}
attrs {
promote: true
}
}
dest_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e"
cas_config_opid_index: 5
I20260805 23:05:32.457811 12085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } }
I20260805 23:05:32.459116 12901 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.461185 13088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:32.461561 12901 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.12.124:42281
I20260805 23:05:32.461737 12901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.461826 12901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:32.462271 12913 master.cc:537] Master@127.3.12.124:42281 shutting down...
I20260805 23:05:32.462754 13088 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Refusing update from remote peer 47cc7c4c2a414f83804aa6688a6f6a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260805 23:05:32.465131 12900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:32.468024 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.460954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.3.12.125:34147)
0805 23:05:32.462737 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.462742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.462911 (+ 169us) server_negotiation.cc:407] Connection header received
0805 23:05:32.463114 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.463117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.463181 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.463253 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:32.463644 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.464505 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.466210 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.466948 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.467032 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.467212 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.467583 (+ 371us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.467666 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684}
W20260805 23:05:32.468446 12061 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e -> Peer 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124:42281): Couldn't send request to peer 504c0d936fa440e8bab92a2cee3d7c40. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260805 23:05:32.468461 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.460708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.0.0.1:46150)
0805 23:05:32.462843 (+ 2135us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.462858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.462981 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.463407 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.463410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.463425 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:32.463521 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.463526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.464654 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.464657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.466016 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.466034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.467315 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.467319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.467417 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.467468 (+ 51us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.468232 (+ 764us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.468335 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2053}
I20260805 23:05:32.469901 12976 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 LEARNER]: Refusing update from remote peer 47cc7c4c2a414f83804aa6688a6f6a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260805 23:05:32.470542 12900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 23:05:32.473057 12976 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.474998 13169 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.475198 13169 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: This master's current role is: LEARNER
I20260805 23:05:32.475405 13015 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47cc7c4c2a414f83804aa6688a6f6a9e. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: NON_VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: true } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.475538 13015 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: This master's current role is: LEARNER
I20260805 23:05:32.476245 12913 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:32.476573 12913 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:32.476677 12913 tablet_replica.cc:330] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: stopping tablet replica
I20260805 23:05:32.477205 12900 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: attempting to promote NON_VOTER eee475496b804e56ac72865c49d6fad0 to VOTER
I20260805 23:05:32.477677 12900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } }
I20260805 23:05:32.479797 12976 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 LEARNER]: Refusing update from remote peer 47cc7c4c2a414f83804aa6688a6f6a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260805 23:05:32.480458 12901 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260805 23:05:32.481299 12061 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e -> Peer 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124:42281): Couldn't send request to peer 504c0d936fa440e8bab92a2cee3d7c40. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260805 23:05:32.483088 12900 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, eee475496b804e56ac72865c49d6fad0 (127.3.12.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.483580 12976 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, eee475496b804e56ac72865c49d6fad0 (127.3.12.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.484680 13171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.484719 12900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.484797 13171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:32.484831 12900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:32.485500 13015 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.485633 13015 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:05:32.494061 12913 master.cc:552] Master@127.3.12.124:42281 shutdown complete.
I20260805 23:05:32.494150 12913 master_runner.cc:305] Clearing existing system tablet
I20260805 23:05:32.494417 12913 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:32.494488 12913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:32.496011 12913 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260805 23:05:32.497192 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.497547 12913 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:32.497678 12913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:32.497718 12913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
uuid: "504c0d936fa440e8bab92a2cee3d7c40"
format_stamp: "Formatted at 2026-08-05 23:05:32 on dist-test-slave-2d19"
server_key: "b9e8ae7fb6856b083a238f76f7e5b355"
server_key_iv: "9764a71498b7e8dc17904eab97e51339"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.499922 12913 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:32.502462 12913 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260805 23:05:32.502564 12913 log.cc:1196] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260805 23:05:32.502831 12913 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Deleting consensus metadata
I20260805 23:05:32.503002 12913 master_runner.cc:315] Copying system tablet from 127.3.12.126:44519
I20260805 23:05:32.504297 12913 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: tablet copy: Beginning tablet copy session from remote peer at address 127.3.12.126:44519
I20260805 23:05:32.514606 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.505040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.3.12.126:44519)
0805 23:05:32.505189 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.505194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.505424 (+ 230us) server_negotiation.cc:407] Connection header received
0805 23:05:32.505561 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.505567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.505639 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.505760 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.506310 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.507191 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.508577 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.508914 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.509027 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.510415 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.510419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.510422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.510453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.512517 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.513124 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.513128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.513130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.513236 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.513595 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.513598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.513600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.513900 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.514000 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.514304 (+ 304us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.514441 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 23:05:32.515069 13179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.504939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60168)
0805 23:05:32.505282 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.505297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.505457 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.505790 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.505793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.506017 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.506156 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.506163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.507348 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.507352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.508447 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.508459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.509048 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.509053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.509172 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.509780 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.509796 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.510252 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.512665 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.512669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.512672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.512972 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.513333 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.513336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.513338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.513465 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.513987 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.513990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.514107 (+ 117us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.514229 (+ 122us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.514397 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":95,"threads_started":1}
I20260805 23:05:32.515575 12118 tablet_copy_service.cc:140] P 47cc7c4c2a414f83804aa6688a6f6a9e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 504c0d936fa440e8bab92a2cee3d7c40 ({username='oryx', principal='oryx/127.3.12.124@KRBTEST.COM'} at 127.0.0.1:60168)
I20260805 23:05:32.515687 12118 tablet_copy_service.cc:161] P 47cc7c4c2a414f83804aa6688a6f6a9e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 504c0d936fa440e8bab92a2cee3d7c40 at {username='oryx', principal='oryx/127.3.12.124@KRBTEST.COM'} at 127.0.0.1:60168: session id = 504c0d936fa440e8bab92a2cee3d7c40-00000000000000000000000000000000
I20260805 23:05:32.516906 12118 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 23:05:32.518599 12913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260805 23:05:32.521670 12913 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: tablet copy: Starting download of 0 data blocks...
I20260805 23:05:32.522012 12913 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: tablet copy: Starting download of 1 WAL segments...
I20260805 23:05:32.523876 12913 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 23:05:32.526026 12118 tablet_copy_service.cc:342] P 47cc7c4c2a414f83804aa6688a6f6a9e: Request end of tablet copy session 504c0d936fa440e8bab92a2cee3d7c40-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.12.124@KRBTEST.COM'} at 127.0.0.1:60168
I20260805 23:05:32.526134 12118 tablet_copy_service.cc:433] P 47cc7c4c2a414f83804aa6688a6f6a9e: ending tablet copy session 504c0d936fa440e8bab92a2cee3d7c40-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 504c0d936fa440e8bab92a2cee3d7c40
I20260805 23:05:32.534432 12913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:32.536329 13183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:32.536387 13180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:32.536825 13181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:32.536880 12913 server_base.cc:1034] running on GCE node
I20260805 23:05:32.537143 12913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:32.537447 12913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:32.538297 12560 heartbeater.cc:503] Master 127.3.12.126:44519 was elected leader, sending a full tablet report...
I20260805 23:05:32.538585 12913 hybrid_clock.cc:648] HybridClock initialized: now 1785971132538560 us; error 43 us; skew 500 ppm
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:32.540236 12913 init.cc:377] Logged in from keytab as oryx/127.3.12.124@KRBTEST.COM (short username oryx)
I20260805 23:05:32.540781 12913 webserver.cc:460] Webserver started at http://127.3.12.124:39673/ using document root <none> and password file <none>
I20260805 23:05:32.540988 12913 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:32.541033 12913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:32.542598 12913 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260805 23:05:32.543773 13199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.544066 12913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260805 23:05:32.544152 12913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
uuid: "504c0d936fa440e8bab92a2cee3d7c40"
format_stamp: "Formatted at 2026-08-05 23:05:32 on dist-test-slave-2d19"
server_key: "b9e8ae7fb6856b083a238f76f7e5b355"
server_key_iv: "9764a71498b7e8dc17904eab97e51339"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.544238 12913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:32.556526 12913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:32.558950 12913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:32.564469 12913 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.124:42281
I20260805 23:05:32.564608 13276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.124:42281 every 8 connection(s)
I20260805 23:05:32.565116 12913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
I20260805 23:05:32.567011 13277 sys_catalog.cc:263] Verifying existing consensus state
I20260805 23:05:32.568146 13277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Bootstrap starting.
I20260805 23:05:32.580921 13277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:05:32.581349 13277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: Bootstrap complete.
I20260805 23:05:32.582059 13277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } }
I20260805 23:05:32.582221 13277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 504c0d936fa440e8bab92a2cee3d7c40, State: Initialized, Role: LEARNER
I20260805 23:05:32.582356 13277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } }
I20260805 23:05:32.582907 13277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:32.583262 13280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.583487 13280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: This master's current role is: LEARNER
I20260805 23:05:32.586331 13299 catalog_manager.cc:1259] Loaded cluster ID: 7281e640834947a28ebb27d1e58020cd
I20260805 23:05:32.586483 13299 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: loading cluster ID for follower catalog manager: success
I20260805 23:05:32.588788 13299 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: acquiring CA information for follower catalog manager: success
I20260805 23:05:32.590273 13299 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260805 23:05:32.599579 13167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:32.599849 13167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:32.599895 13167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:32.604419 13167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:32.604485 13167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:32.604583 13167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:32.610042 13167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab
--principal=oryx/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:37743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=37387
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147,127.3.12.124:42281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:32.611150 13167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:32.612366 13167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:32.623039 13309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:32.623063 13311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:32.623209 13167 server_base.cc:1034] running on GCE node
W20260805 23:05:32.623008 13308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:32.623675 13167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:32.624346 13167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:32.625520 13167 hybrid_clock.cc:648] HybridClock initialized: now 1785971132625482 us; error 56 us; skew 500 ppm
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:32.628937 13167 init.cc:377] Logged in from keytab as oryx/127.3.12.65@KRBTEST.COM (short username oryx)
I20260805 23:05:32.629827 13167 webserver.cc:460] Webserver started at http://127.3.12.65:37387/ using document root <none> and password file <none>
I20260805 23:05:32.630384 13167 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:32.630463 13167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:32.634493 13167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:05:32.637005 13325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.638360 13167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:32.638512 13167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "a616b04439d245d4b24b4b0f3c164363"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "3c8d506ccc56e4968dd3cd7624fdb681"
server_key_iv: "4875d992abaa8e5f47c072b14e57521e"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.639094 13167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:32.658627 13167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:32.662380 13167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:32.662603 13167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:32.663259 13167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:32.664198 13167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:32.664247 13167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.664382 13167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:32.664415 13167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.674253 13167 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:37743
I20260805 23:05:32.674373 13487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:37743 every 8 connection(s)
I20260805 23:05:32.675436 13167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:32.684458 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 13167
I20260805 23:05:32.684680 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 12375
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.125@KRBTEST.COM
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.124@KRBTEST.COM
I20260805 23:05:32.694402 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.677448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:53143 (local address 127.3.12.126:44519)
0805 23:05:32.677617 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.677623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.678846 (+ 1223us) server_negotiation.cc:407] Connection header received
0805 23:05:32.680239 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.680245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.680333 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.680409 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.681866 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.682691 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.684397 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.684756 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.684879 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.689001 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.689007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.689013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.689047 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.690761 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.691427 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.691433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.691438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.691812 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.692846 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.692852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.692855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.693134 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.693221 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.694021 (+ 800us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.694199 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260805 23:05:32.695350 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.677558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:46767 (local address 127.3.12.125:34147)
0805 23:05:32.677678 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.677681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.678837 (+ 1156us) server_negotiation.cc:407] Connection header received
0805 23:05:32.680150 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.680153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.680218 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.680300 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.681866 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.682691 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.684394 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.684744 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.684837 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.689623 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.689631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.689637 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.689678 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.692708 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.693400 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.693407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.693412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.693537 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.693970 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.693976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.693981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.694317 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.694418 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.694913 (+ 495us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.695105 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260805 23:05:32.696549 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.678035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.3.12.65:46767)
0805 23:05:32.678646 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.678714 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.679900 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.680658 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.680665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.681218 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.681710 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.681718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.682968 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.682971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.684223 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.684232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.685038 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.685042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.685376 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.686543 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.686560 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.689414 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.692868 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.692874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.692879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.693242 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.693687 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.693693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.693698 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.693828 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.694512 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.694518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.694651 (+ 133us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.695532 (+ 881us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.695742 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":452,"spinlock_wait_cycles":68096,"thread_start_us":156,"threads_started":1}
I20260805 23:05:32.696535 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.678035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.65:53143)
0805 23:05:32.678646 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.678714 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.679900 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.680658 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.680665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.681236 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.681709 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.681718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.682799 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.682802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.684199 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.684213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.685070 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.685085 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.685378 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.686224 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.686255 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.688779 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.690889 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.690894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.690908 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.691294 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.692217 (+ 923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.692220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.692222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.692462 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.693322 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.693331 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.693704 (+ 373us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.694673 (+ 969us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.695193 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":430,"spinlock_wait_cycles":35584,"thread_start_us":164,"threads_started":1}
I20260805 23:05:32.697340 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.679330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:35423 (local address 127.3.12.124:42281)
0805 23:05:32.679572 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.679584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.679602 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:05:32.680155 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.680159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.680230 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.680382 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.681853 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.682565 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.684397 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.684770 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.684868 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.691525 (+ 6657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.691531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.691535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.691579 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.693424 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.693911 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.693918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.693920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.694014 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.694321 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.694324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.694326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.694571 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.694665 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.696670 (+ 2005us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.697178 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260805 23:05:32.696534 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.679213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42281 (local address 127.3.12.65:35423)
0805 23:05:32.679536 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.679550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.679911 (+ 361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.680656 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.680663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.681177 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.681693 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.681707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.682719 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.682724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.684222 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.684232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.685038 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.685042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.685373 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.686371 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.686392 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.691348 (+ 4956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.693540 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.693543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.693545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.693785 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.694122 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.694125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.694126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.694215 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.694711 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.694714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.694815 (+ 101us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.694910 (+ 95us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.695140 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":262,"spinlock_wait_cycles":1280,"thread_start_us":91,"threads_started":1}
I20260805 23:05:32.698715 13488 heartbeater.cc:347] Connected to a master server at 127.3.12.124:42281
I20260805 23:05:32.698966 13488 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:32.699492 13488 heartbeater.cc:511] Master 127.3.12.124:42281 requested a full tablet report, sending...
I20260805 23:05:32.701320 13228 ts_manager.cc:194] Registered new tserver with Master: a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743)
I20260805 23:05:32.705681 13493 heartbeater.cc:347] Connected to a master server at 127.3.12.125:34147
I20260805 23:05:32.705789 13493 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:32.706113 13493 heartbeater.cc:511] Master 127.3.12.125:34147 requested a full tablet report, sending...
I20260805 23:05:32.706477 13492 heartbeater.cc:347] Connected to a master server at 127.3.12.126:44519
I20260805 23:05:32.706559 13492 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:32.706740 13492 heartbeater.cc:511] Master 127.3.12.126:44519 requested a full tablet report, sending...
I20260805 23:05:32.707408 12966 ts_manager.cc:194] Registered new tserver with Master: a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743)
I20260805 23:05:32.707581 12085 ts_manager.cc:194] Re-registered known tserver with Master: a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743)
I20260805 23:05:32.708426 12085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:53143
I20260805 23:05:32.712126 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:40099
--local_ip_for_outbound_sockets=127.3.12.66
--tserver_master_addrs=127.3.12.126:44519
--webserver_port=38475
--webserver_interface=127.3.12.66
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab
--principal=oryx/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147,127.3.12.124:42281 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:05:32.761327 12746 heartbeater.cc:503] Master 127.3.12.126:44519 was elected leader, sending a full tablet report...
W20260805 23:05:32.834154 13501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:32.834474 13501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:32.834550 13501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:32.839084 13501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:32.839188 13501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:32.839282 13501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:32.845834 13501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab
--principal=oryx/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:40099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=38475
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147,127.3.12.124:42281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:32.847126 13501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:32.848366 13501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:05:32.859810 13501 server_base.cc:1034] running on GCE node
W20260805 23:05:32.859725 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:32.859725 13513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:32.859771 13510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:32.860469 13501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:32.861227 13501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:32.862480 13501 hybrid_clock.cc:648] HybridClock initialized: now 1785971132862453 us; error 40 us; skew 500 ppm
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:32.865909 13501 init.cc:377] Logged in from keytab as oryx/127.3.12.66@KRBTEST.COM (short username oryx)
I20260805 23:05:32.866875 13501 webserver.cc:460] Webserver started at http://127.3.12.66:38475/ using document root <none> and password file <none>
I20260805 23:05:32.867517 13501 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:32.867636 13501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:32.872382 13501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:05:32.875041 13524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.876518 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.872550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.3.12.124:42281)
0805 23:05:32.872673 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.872676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.872692 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:05:32.872883 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.872887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.872958 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.873040 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:32.873367 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.873898 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.875311 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.875944 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.876034 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.876152 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.876350 (+ 198us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.876413 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260805 23:05:32.876576 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.872418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42281 (local address 127.0.0.1:36420)
0805 23:05:32.872596 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.872615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.872739 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.873090 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.873093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.873116 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:32.873234 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.873242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.874025 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.874029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.875140 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.875153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.876079 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.876084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.876185 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.876215 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.876330 (+ 115us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.876433 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260805 23:05:32.876785 13501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:32.877264 13233 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260805 23:05:32.876951 13501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "b40a7bc2fdef4f0f808206ad3eeec5be"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "cd4bfc14a08c4567847c07c3dad56321"
server_key_iv: "d65c98bbd7fdee90d3c413337c1eda40"
server_key_version: "encryptionkey@0"
I20260805 23:05:32.877650 13280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47cc7c4c2a414f83804aa6688a6f6a9e. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:32.877624 13501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:32.877805 13280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: This master's current role is: LEARNER
I20260805 23:05:32.896389 13501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:32.899606 13501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:32.899817 13501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:32.900532 13501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:32.901453 13501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:32.901528 13501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.901600 13501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:32.901651 13501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:32.911154 13501 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:40099
I20260805 23:05:32.911175 13687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:40099 every 8 connection(s)
I20260805 23:05:32.912231 13501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:32.919688 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 13501
I20260805 23:05:32.919867 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 12566
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.125@KRBTEST.COM
Aug 05 23:05:32 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971132, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.124@KRBTEST.COM
I20260805 23:05:32.929991 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.914823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:46447 (local address 127.3.12.126:44519)
0805 23:05:32.914965 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.914970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.915325 (+ 355us) server_negotiation.cc:407] Connection header received
0805 23:05:32.916635 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.916639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.916693 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.916748 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.918145 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.918941 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920390 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.920721 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920800 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.924820 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.924824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.924827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.924858 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.926751 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.927954 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.927959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.927964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.928076 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.928563 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.928568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.928570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.928825 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.928914 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.929657 (+ 743us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.929793 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:05:32.930523 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.915864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:53939 (local address 127.3.12.124:42281)
0805 23:05:32.915993 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.915997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.916100 (+ 103us) server_negotiation.cc:407] Connection header received
0805 23:05:32.916617 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.916621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.916680 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.916757 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.918122 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.918664 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920067 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.920501 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920610 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.926544 (+ 5934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.926560 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.926565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.926600 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.928649 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.929124 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.929128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.929129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.929214 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.929541 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.929545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.929561 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.929784 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.929856 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.930233 (+ 377us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.930372 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:05:32.931469 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.914223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:35805 (local address 127.3.12.125:34147)
0805 23:05:32.914408 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:32.914415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:32.915323 (+ 908us) server_negotiation.cc:407] Connection header received
0805 23:05:32.916635 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:32.916639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:32.916693 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:32.916748 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:32.918140 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.918943 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920295 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.920634 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920707 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.926633 (+ 5926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:32.926641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:32.926647 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:32.926695 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:32.928834 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.929510 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.929517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.929523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.929644 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:32.930087 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:32.930094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:32.930098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:32.930422 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:32.930530 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:32.931026 (+ 496us) server_negotiation.cc:299] Negotiation successful
0805 23:05:32.931200 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 23:05:32.932430 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.914611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:46447)
0805 23:05:32.915163 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.915225 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.916406 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.916932 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.916940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.917393 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.917984 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.917993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.919092 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.919095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.920271 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.920282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.920961 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.921280 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.922270 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.922289 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.924649 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.927398 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.927407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.927428 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.927824 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.928203 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.928209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.928213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.928436 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.928987 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.928994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.929365 (+ 371us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.930539 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.931223 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":317,"spinlock_wait_cycles":31104,"thread_start_us":139,"threads_started":1}
I20260805 23:05:32.932448 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.914611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.3.12.66:35805)
0805 23:05:32.915163 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.915225 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.916406 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.916932 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.916940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.917390 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.917984 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.917993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.919092 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.919095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.920136 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.920147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.920870 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.921240 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.922138 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.922155 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.926401 (+ 4246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.928992 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.928999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.929004 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.929343 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.929797 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.929803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.929808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.929932 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.930627 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.930633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.930772 (+ 139us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.931654 (+ 882us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.931860 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":340,"spinlock_wait_cycles":81536,"thread_start_us":149,"threads_started":1}
I20260805 23:05:32.932723 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:32.915732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42281 (local address 127.3.12.66:53939)
0805 23:05:32.916010 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:32.916022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:32.916404 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:32.916931 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:32.916938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:32.917387 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:32.917968 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.917982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.918811 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.918816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.919921 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:32.919935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:32.920868 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:32.920877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:32.921293 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:32.922121 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:32.922151 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:32.926354 (+ 4203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:32.928766 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.928769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.928771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.929012 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.929326 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:32.929329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:32.929331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:32.929430 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:32.929930 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:32.929933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:32.930025 (+ 92us) client_negotiation.cc:769] Sending connection context
0805 23:05:32.930744 (+ 719us) client_negotiation.cc:240] Negotiation successful
0805 23:05:32.931275 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":333,"spinlock_wait_cycles":12672,"thread_start_us":81,"threads_started":1}
I20260805 23:05:32.934216 13689 heartbeater.cc:347] Connected to a master server at 127.3.12.126:44519
I20260805 23:05:32.934491 13689 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:32.935016 13689 heartbeater.cc:511] Master 127.3.12.126:44519 requested a full tablet report, sending...
I20260805 23:05:32.936146 12085 ts_manager.cc:194] Re-registered known tserver with Master: b40a7bc2fdef4f0f808206ad3eeec5be (127.3.12.66:40099)
I20260805 23:05:32.937005 12085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:46447
I20260805 23:05:32.941964 13688 heartbeater.cc:347] Connected to a master server at 127.3.12.124:42281
I20260805 23:05:32.942095 13688 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:32.942341 13688 heartbeater.cc:511] Master 127.3.12.124:42281 requested a full tablet report, sending...
I20260805 23:05:32.942593 13691 heartbeater.cc:347] Connected to a master server at 127.3.12.125:34147
I20260805 23:05:32.942687 13691 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:32.942831 13691 heartbeater.cc:511] Master 127.3.12.125:34147 requested a full tablet report, sending...
I20260805 23:05:32.943228 13228 ts_manager.cc:194] Registered new tserver with Master: b40a7bc2fdef4f0f808206ad3eeec5be (127.3.12.66:40099)
I20260805 23:05:32.943969 12966 ts_manager.cc:194] Registered new tserver with Master: b40a7bc2fdef4f0f808206ad3eeec5be (127.3.12.66:40099)
I20260805 23:05:32.947758 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:46457
--local_ip_for_outbound_sockets=127.3.12.67
--tserver_master_addrs=127.3.12.126:44519
--webserver_port=45447
--webserver_interface=127.3.12.67
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab
--principal=oryx/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147,127.3.12.124:42281 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:33.069830 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:33.070111 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:33.070199 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:33.074776 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:33.074841 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:33.074934 13697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:33.080421 13697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab
--principal=oryx/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:46457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=45447
--tserver_master_addrs=127.3.12.126:44519,127.3.12.125:34147,127.3.12.124:42281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:33.081511 13697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:33.082589 13697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:33.092744 13705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:33.092881 13697 server_base.cc:1034] running on GCE node
W20260805 23:05:33.092769 13708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:33.092769 13706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:33.093591 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:33.094218 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:33.095415 13697 hybrid_clock.cc:648] HybridClock initialized: now 1785971133095397 us; error 29 us; skew 500 ppm
Aug 05 23:05:33 dist-test-slave-2d19 krb5kdc[12022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971133, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:33.098263 13697 init.cc:377] Logged in from keytab as oryx/127.3.12.67@KRBTEST.COM (short username oryx)
I20260805 23:05:33.099164 13697 webserver.cc:460] Webserver started at http://127.3.12.67:45447/ using document root <none> and password file <none>
I20260805 23:05:33.099745 13697 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:33.099862 13697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:33.103724 13697 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:33.105799 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:33.106981 13697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:33.107141 13697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "6f097662686240abbf18cd043df2a7f5"
format_stamp: "Formatted at 2026-08-05 23:05:31 on dist-test-slave-2d19"
server_key: "4243acd7b536c459d67b369409836cce"
server_key_iv: "61296f7c42d382c687dd42c11230fc55"
server_key_version: "encryptionkey@0"
I20260805 23:05:33.107712 13697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:33.119015 13697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:33.121833 13697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:33.122037 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:33.122604 13697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:33.123443 13697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:33.123513 13697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:33.123585 13697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:33.123638 13697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:33.133478 13697 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:46457
I20260805 23:05:33.133528 13893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:46457 every 8 connection(s)
I20260805 23:05:33.134513 13697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:33.135051 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 13697
Aug 05 23:05:33 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971133, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.124@KRBTEST.COM
I20260805 23:05:33.149886 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.137515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.3.12.125:34147)
0805 23:05:33.137650 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.137656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.138824 (+ 1168us) server_negotiation.cc:407] Connection header received
0805 23:05:33.139014 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.139017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.139068 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.139122 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.140115 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.140920 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.142147 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.142515 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.142582 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.144884 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.144888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.144890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.144913 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.147310 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.148103 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.148108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.148112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.148215 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.148587 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.148594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.148596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.149088 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.149195 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.149564 (+ 369us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.149683 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:05:33.150674 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.138166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.3.12.124:42281)
0805 23:05:33.138417 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.138421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.138733 (+ 312us) server_negotiation.cc:407] Connection header received
0805 23:05:33.138797 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.138803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.138864 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.138970 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.140614 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.141364 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.142460 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.142788 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.142878 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.144879 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.144883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.144886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.144913 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.147135 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.147683 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.147686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.147688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.147785 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.148966 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.148969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.148971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.149179 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.149294 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.150365 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.150513 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
Aug 05 23:05:33 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971133, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.125@KRBTEST.COM
Aug 05 23:05:33 dist-test-slave-2d19 krb5kdc[12022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971133, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:33.153894 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.137907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.3.12.126:44519)
0805 23:05:33.138230 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.138235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.140781 (+ 2546us) server_negotiation.cc:407] Connection header received
0805 23:05:33.141579 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.141582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.141625 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.141684 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.142649 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.143950 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.145115 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.145942 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.146013 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.148791 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.148814 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.148817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.148843 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.150758 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.152377 (+ 1619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.152383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.152385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.152497 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.152838 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.152842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.152843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.153095 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.153224 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.153587 (+ 363us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.153718 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":71,"threads_started":1}
I20260805 23:05:33.156560 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.137157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:36927 (local address 127.3.12.124:42281)
0805 23:05:33.137292 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.137296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.138984 (+ 1688us) server_negotiation.cc:407] Connection header received
0805 23:05:33.139909 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.139912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.139976 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.140068 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.142044 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.142814 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.144650 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.144997 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.145080 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.151241 (+ 6161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.151245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.151247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.151275 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.153441 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.154115 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.154120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.154122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.154208 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.154680 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.154684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.154687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.154929 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.155035 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.156239 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.156400 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:05:33.158154 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.137166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42281 (local address 127.3.12.67:36927)
0805 23:05:33.138901 (+ 1735us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:33.138914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:33.139657 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:33.140241 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:33.140250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:33.140922 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:33.141887 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.141898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.142995 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.142998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.144480 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.144490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.145276 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.145281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.145517 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.148387 (+ 2870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:33.148407 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:33.151043 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:33.153558 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:33.153567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:33.153586 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:33.153984 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:33.154339 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:33.154345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:33.154346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:33.154557 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:33.155112 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:33.155120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:33.155451 (+ 331us) client_negotiation.cc:769] Sending connection context
0805 23:05:33.156722 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0805 23:05:33.157072 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1646,"mutex_wait_us":455,"spinlock_wait_cycles":15488,"thread_start_us":153,"threads_started":1}
I20260805 23:05:33.158172 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.136780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:36615 (local address 127.3.12.126:44519)
0805 23:05:33.137023 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.137028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.138670 (+ 1642us) server_negotiation.cc:407] Connection header received
0805 23:05:33.140179 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.140182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.140231 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.140329 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.146505 (+ 6176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.147255 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.150557 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.150898 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.151014 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.154345 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.154348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.154351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.154375 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.156108 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.156903 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.156909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.156914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.157003 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.157298 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.157302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.157303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.157519 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.157601 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.157933 (+ 332us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.158043 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260805 23:05:33.158484 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.138210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.3.12.67:42403)
0805 23:05:33.139319 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:33.139332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:33.140004 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:33.140469 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:33.140471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:33.140930 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:33.145750 (+ 4820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.145758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.146744 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.146747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.147597 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.147609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.148373 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.148377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.148475 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.151623 (+ 3148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:33.151641 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:33.153309 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:33.155846 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:33.155849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:33.155850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:33.156101 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:33.156450 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:33.156454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:33.156456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:33.156551 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:33.157025 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:33.157031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:33.157138 (+ 107us) client_negotiation.cc:769] Sending connection context
0805 23:05:33.157245 (+ 107us) client_negotiation.cc:240] Negotiation successful
0805 23:05:33.157385 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":4460,"spinlock_wait_cycles":8960,"thread_start_us":86,"threads_started":1}
I20260805 23:05:33.159092 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.138551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:42403 (local address 127.3.12.125:34147)
0805 23:05:33.139171 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.139175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.139992 (+ 817us) server_negotiation.cc:407] Connection header received
0805 23:05:33.140115 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.140118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.140162 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.140239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.145881 (+ 5642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.146620 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.147760 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.148107 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.148180 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.153483 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.153489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.153494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.153525 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.155724 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.156219 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.156222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.156224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.156334 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.156640 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.156647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.156650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.156871 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.156977 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.158794 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.158917 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":96,"threads_started":1}
I20260805 23:05:33.159529 13894 heartbeater.cc:347] Connected to a master server at 127.3.12.124:42281
I20260805 23:05:33.159657 13896 heartbeater.cc:347] Connected to a master server at 127.3.12.125:34147
I20260805 23:05:33.159758 13894 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:33.159759 13896 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:33.159885 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.137166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:36615)
0805 23:05:33.138492 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:33.138550 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:33.139657 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:33.140389 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:33.140394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:33.141334 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:33.141682 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.141702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.147773 (+ 6071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.147776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.150387 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.150399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.151161 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.151165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.151259 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.151743 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:33.151760 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:33.154188 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:33.156271 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:33.156277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:33.156281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:33.156579 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:33.157107 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:33.157110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:33.157112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:33.157195 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:33.157659 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:33.157662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:33.157744 (+ 82us) client_negotiation.cc:769] Sending connection context
0805 23:05:33.158907 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0805 23:05:33.159037 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":2,"spinlock_wait_cycles":443136,"thread_start_us":122,"threads_started":1}
I20260805 23:05:33.160400 13896 heartbeater.cc:511] Master 127.3.12.125:34147 requested a full tablet report, sending...
I20260805 23:05:33.160413 13894 heartbeater.cc:511] Master 127.3.12.124:42281 requested a full tablet report, sending...
I20260805 23:05:33.160748 13895 heartbeater.cc:347] Connected to a master server at 127.3.12.126:44519
I20260805 23:05:33.160827 13895 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:33.160979 13895 heartbeater.cc:511] Master 127.3.12.126:44519 requested a full tablet report, sending...
I20260805 23:05:33.157737 12085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260805 23:05:33.161590 12080 ts_manager.cc:194] Re-registered known tserver with Master: 6f097662686240abbf18cd043df2a7f5 (127.3.12.67:46457)
I20260805 23:05:33.161650 13228 ts_manager.cc:194] Registered new tserver with Master: 6f097662686240abbf18cd043df2a7f5 (127.3.12.67:46457)
I20260805 23:05:33.162122 12966 ts_manager.cc:194] Registered new tserver with Master: 6f097662686240abbf18cd043df2a7f5 (127.3.12.67:46457)
I20260805 23:05:33.162317 12080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:36615
W20260805 23:05:33.162389 12085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:33.165095 12901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: NON_VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: true } } }
I20260805 23:05:33.165202 12901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:33.168792 13930 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: attempting to promote NON_VOTER 504c0d936fa440e8bab92a2cee3d7c40 to VOTER
I20260805 23:05:33.169155 13930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } }
I20260805 23:05:33.170629 13233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 LEARNER]: Refusing update from remote peer 47cc7c4c2a414f83804aa6688a6f6a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260805 23:05:33.171013 12901 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260805 23:05:33.171717 12976 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Refusing update from remote peer 47cc7c4c2a414f83804aa6688a6f6a9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260805 23:05:33.172147 13931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260805 23:05:33.173516 13930 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:33.173978 12976 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:33.174103 13233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:33.175078 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:33.175168 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:33.175282 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:33.175376 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:33.175484 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:33.175587 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:33.176550 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:33.176645 13280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:33.176767 13280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:05:33.177428 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:05:33.184122 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.178430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:46457 (local address 127.0.0.1:39352)
0805 23:05:33.178560 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:33.178577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:33.178660 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:33.179509 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:33.179512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:33.179532 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:33.179647 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.179653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.181472 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.181475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.182568 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.182579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.183710 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.183714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.183800 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.183824 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 23:05:33.183892 (+ 68us) client_negotiation.cc:240] Negotiation successful
0805 23:05:33.183993 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260805 23:05:33.184312 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.178609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.3.12.67:46457)
0805 23:05:33.178969 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.178985 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.179009 (+ 24us) server_negotiation.cc:407] Connection header received
0805 23:05:33.179102 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.179110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.179340 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.179811 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:33.179959 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.181333 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.182757 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.183567 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.183712 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.183870 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.183922 (+ 52us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.184096 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":119,"threads_started":1}
I20260805 23:05:33.185775 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.179046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:37743 (local address 127.0.0.1:59882)
0805 23:05:33.179285 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:33.179300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:33.179382 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:33.180231 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:33.180235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:33.180275 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:33.180367 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.180372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.182646 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.182650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.183905 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.183916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.185371 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.185375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.185506 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.185532 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 23:05:33.185580 (+ 48us) client_negotiation.cc:240] Negotiation successful
0805 23:05:33.185646 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260805 23:05:33.186564 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.179479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.3.12.65:37743)
0805 23:05:33.179773 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.179788 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.179811 (+ 23us) server_negotiation.cc:407] Connection header received
0805 23:05:33.179882 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.179887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.180080 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.180505 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:33.180628 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.182520 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.184069 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.185242 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.186061 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.186250 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.186298 (+ 48us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.186382 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260805 23:05:33.187882 13796 tablet_service.cc:1467] Processing CreateTablet for tablet c35f8eb504f048a1a56184b7415ec967 (DEFAULT_TABLE table=test-table [id=324268a356194a86a7f30dea743ed015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:33.189085 13796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35f8eb504f048a1a56184b7415ec967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:33.189180 13389 tablet_service.cc:1467] Processing CreateTablet for tablet c35f8eb504f048a1a56184b7415ec967 (DEFAULT_TABLE table=test-table [id=324268a356194a86a7f30dea743ed015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:33.190027 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.179871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:40099 (local address 127.0.0.1:37352)
0805 23:05:33.184353 (+ 4482us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:33.184369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:33.184454 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:33.185627 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:33.185631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:33.185652 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:33.185729 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.185735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.187376 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.187380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.188483 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.188494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.189672 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.189676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.189760 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.189784 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 23:05:33.189832 (+ 48us) client_negotiation.cc:240] Negotiation successful
0805 23:05:33.189895 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4416,"thread_start_us":91,"threads_started":1}
I20260805 23:05:33.190044 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35f8eb504f048a1a56184b7415ec967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:33.190737 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.180427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.3.12.66:40099)
0805 23:05:33.182980 (+ 2553us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.182993 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.185136 (+ 2143us) server_negotiation.cc:407] Connection header received
0805 23:05:33.185220 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.185225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.185468 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.185874 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:33.186022 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.187247 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.188743 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.189520 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.190303 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.190442 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.190492 (+ 50us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.190579 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2389,"thread_start_us":112,"threads_started":1}
I20260805 23:05:33.193370 13595 tablet_service.cc:1467] Processing CreateTablet for tablet c35f8eb504f048a1a56184b7415ec967 (DEFAULT_TABLE table=test-table [id=324268a356194a86a7f30dea743ed015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:33.194372 13595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35f8eb504f048a1a56184b7415ec967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:33.198145 13952 tablet_bootstrap.cc:492] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: Bootstrap starting.
I20260805 23:05:33.200003 13952 tablet_bootstrap.cc:654] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:33.200474 13951 tablet_bootstrap.cc:492] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: Bootstrap starting.
I20260805 23:05:33.200909 13952 log.cc:824] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:33.202040 13953 tablet_bootstrap.cc:492] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: Bootstrap starting.
I20260805 23:05:33.202494 13951 tablet_bootstrap.cc:654] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:33.203146 13951 log.cc:824] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:33.203943 13953 tablet_bootstrap.cc:654] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:33.204669 13953 log.cc:824] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:33.205161 13951 tablet_bootstrap.cc:492] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: No bootstrap required, opened a new log
I20260805 23:05:33.205161 13952 tablet_bootstrap.cc:492] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: No bootstrap required, opened a new log
I20260805 23:05:33.205353 13952 ts_tablet_manager.cc:1397] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260805 23:05:33.205353 13951 ts_tablet_manager.cc:1397] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 23:05:33.206367 13953 tablet_bootstrap.cc:492] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: No bootstrap required, opened a new log
I20260805 23:05:33.206562 13953 ts_tablet_manager.cc:1397] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260805 23:05:33.209321 13952 raft_consensus.cc:348] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.209607 13952 raft_consensus.cc:374] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:33.209671 13952 raft_consensus.cc:729] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a616b04439d245d4b24b4b0f3c164363, State: Initialized, Role: FOLLOWER
I20260805 23:05:33.209869 13951 raft_consensus.cc:348] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.210114 13951 raft_consensus.cc:374] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:33.210172 13951 raft_consensus.cc:729] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f097662686240abbf18cd043df2a7f5, State: Initialized, Role: FOLLOWER
I20260805 23:05:33.210115 13952 consensus_queue.cc:260] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.210130 13953 raft_consensus.cc:348] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.210397 13953 raft_consensus.cc:374] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:33.210458 13953 raft_consensus.cc:729] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a7bc2fdef4f0f808206ad3eeec5be, State: Initialized, Role: FOLLOWER
I20260805 23:05:33.210682 13951 consensus_queue.cc:260] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.210903 13953 consensus_queue.cc:260] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.211130 13952 ts_tablet_manager.cc:1428] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 23:05:33.211798 13951 ts_tablet_manager.cc:1428] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 23:05:33.211841 13953 ts_tablet_manager.cc:1428] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260805 23:05:33.212123 13492 heartbeater.cc:503] Master 127.3.12.126:44519 was elected leader, sending a full tablet report...
I20260805 23:05:33.212992 13895 heartbeater.cc:503] Master 127.3.12.126:44519 was elected leader, sending a full tablet report...
I20260805 23:05:33.213992 13689 heartbeater.cc:503] Master 127.3.12.126:44519 was elected leader, sending a full tablet report...
I20260805 23:05:33.226173 13964 raft_consensus.cc:482] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:33.226382 13964 raft_consensus.cc:504] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.227519 13964 leader_election.cc:288] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f097662686240abbf18cd043df2a7f5 (127.3.12.67:46457), a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743)
I20260805 23:05:33.233469 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.227713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:46457 (local address 127.3.12.66:38549)
0805 23:05:33.227919 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:33.227934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:33.228024 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:33.228507 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:33.228704 (+ 197us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:33.228719 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:33.228881 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.228888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.229942 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.229947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.231612 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.231626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.232880 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.232887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.233037 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.233068 (+ 31us) client_negotiation.cc:769] Sending connection context
0805 23:05:33.233153 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 23:05:33.233255 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":127}
I20260805 23:05:33.233518 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.227928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:38549 (local address 127.3.12.67:46457)
0805 23:05:33.228198 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.228204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.228219 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:33.228296 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.228302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.228357 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.228578 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:33.229014 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.229799 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.231770 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.232564 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.232687 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.232809 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.233179 (+ 370us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.233323 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260805 23:05:33.234012 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.228352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:37743 (local address 127.3.12.66:56481)
0805 23:05:33.228522 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:33.228536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:33.228603 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:33.229228 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:33.229231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:33.229249 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:33.229343 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.229348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.230895 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.230898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.232437 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:33.232449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.233553 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.233558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:33.233671 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.233699 (+ 28us) client_negotiation.cc:769] Sending connection context
0805 23:05:33.233776 (+ 77us) client_negotiation.cc:240] Negotiation successful
0805 23:05:33.233851 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":32}
I20260805 23:05:33.234285 13826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35f8eb504f048a1a56184b7415ec967" candidate_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f097662686240abbf18cd043df2a7f5" is_pre_election: true
I20260805 23:05:33.234668 13826 raft_consensus.cc:2454] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a7bc2fdef4f0f808206ad3eeec5be in term 0.
I20260805 23:05:33.234956 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.228720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:56481 (local address 127.3.12.65:37743)
0805 23:05:33.228914 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.228921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.228937 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:05:33.228990 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.228994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.229045 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.229123 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:33.229608 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.230689 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.232580 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.233311 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.233420 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.233488 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.234737 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.234832 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 23:05:33.235347 13527 leader_election.cc:302] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f097662686240abbf18cd043df2a7f5, b40a7bc2fdef4f0f808206ad3eeec5be; no voters:
I20260805 23:05:33.235647 13964 raft_consensus.cc:2781] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:33.235741 13964 raft_consensus.cc:482] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:33.235798 13964 raft_consensus.cc:3037] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:33.235803 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35f8eb504f048a1a56184b7415ec967" candidate_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a616b04439d245d4b24b4b0f3c164363" is_pre_election: true
I20260805 23:05:33.236121 13422 raft_consensus.cc:2454] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40a7bc2fdef4f0f808206ad3eeec5be in term 0.
I20260805 23:05:33.237284 13964 raft_consensus.cc:504] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.237689 13964 leader_election.cc:288] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [CANDIDATE]: Term 1 election: Requested vote from peers 6f097662686240abbf18cd043df2a7f5 (127.3.12.67:46457), a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743)
I20260805 23:05:33.238018 13826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35f8eb504f048a1a56184b7415ec967" candidate_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f097662686240abbf18cd043df2a7f5"
I20260805 23:05:33.238142 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35f8eb504f048a1a56184b7415ec967" candidate_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a616b04439d245d4b24b4b0f3c164363"
I20260805 23:05:33.238176 13826 raft_consensus.cc:3037] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:33.238250 13422 raft_consensus.cc:3037] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:33.239225 13826 raft_consensus.cc:2454] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a7bc2fdef4f0f808206ad3eeec5be in term 1.
I20260805 23:05:33.239230 13422 raft_consensus.cc:2454] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40a7bc2fdef4f0f808206ad3eeec5be in term 1.
I20260805 23:05:33.239513 13527 leader_election.cc:302] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f097662686240abbf18cd043df2a7f5, b40a7bc2fdef4f0f808206ad3eeec5be; no voters:
I20260805 23:05:33.239691 13964 raft_consensus.cc:2781] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:33.239938 13964 raft_consensus.cc:686] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 LEADER]: Becoming Leader. State: Replica: b40a7bc2fdef4f0f808206ad3eeec5be, State: Running, Role: LEADER
I20260805 23:05:33.240317 13964 consensus_queue.cc:237] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } }
I20260805 23:05:33.243839 12080 catalog_manager.cc:5495] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be reported cstate change: term changed from 0 to 1, leader changed from <none> to b40a7bc2fdef4f0f808206ad3eeec5be (127.3.12.66). New cstate: current_term: 1 leader_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:33.262554 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.258236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.3.12.66:40099)
0805 23:05:33.258655 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.258658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.258668 (+ 10us) server_negotiation.cc:407] Connection header received
0805 23:05:33.258759 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.258762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.258832 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.258933 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:33.259270 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.259775 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.260886 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.261115 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.261248 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.261565 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:33.262060 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:33.262164 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.262371 (+ 207us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.262445 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260805 23:05:33.268278 13826 raft_consensus.cc:1261] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 1 FOLLOWER]: Refusing update from remote peer b40a7bc2fdef4f0f808206ad3eeec5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:33.268301 13422 raft_consensus.cc:1261] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 1 FOLLOWER]: Refusing update from remote peer b40a7bc2fdef4f0f808206ad3eeec5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:33.269034 13964 consensus_queue.cc:1035] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:33.269253 13966 consensus_queue.cc:1035] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:33.280108 13972 mvcc.cc:205] Tried to move back new op lower bound from 7315337761859686400 to 7315337761755119616. Current Snapshot: MvccSnapshot[applied={T|T < 7315337761859686400}]
I20260805 23:05:33.290710 12080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:33.290894 12080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:33.295250 12080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47cc7c4c2a414f83804aa6688a6f6a9e: Sending DeleteTablet for 3 replicas of tablet c35f8eb504f048a1a56184b7415ec967
I20260805 23:05:33.296200 13389 tablet_service.cc:1514] Processing DeleteTablet for tablet c35f8eb504f048a1a56184b7415ec967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:33 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:59882
I20260805 23:05:33.296386 13796 tablet_service.cc:1514] Processing DeleteTablet for tablet c35f8eb504f048a1a56184b7415ec967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:33 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:39352
I20260805 23:05:33.296386 13595 tablet_service.cc:1514] Processing DeleteTablet for tablet c35f8eb504f048a1a56184b7415ec967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:33 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:37352
I20260805 23:05:33.296881 13985 tablet_replica.cc:330] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: stopping tablet replica
I20260805 23:05:33.297220 13985 raft_consensus.cc:2229] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:33.297540 13985 raft_consensus.cc:2258] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:33.299101 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 12038
I20260805 23:05:33.305096 13985 ts_tablet_manager.cc:1905] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:33.307962 13985 ts_tablet_manager.cc:1918] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:33.308077 13985 log.cc:1196] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/c35f8eb504f048a1a56184b7415ec967
I20260805 23:05:33.308466 13985 ts_tablet_manager.cc:1939] T c35f8eb504f048a1a56184b7415ec967 P 6f097662686240abbf18cd043df2a7f5: Deleting consensus metadata
I20260805 23:05:33.309911 13983 tablet_replica.cc:330] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: stopping tablet replica
I20260805 23:05:33.310249 13983 raft_consensus.cc:2229] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:33.310657 13983 raft_consensus.cc:2258] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:33.311570 13982 tablet_replica.cc:330] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: stopping tablet replica
I20260805 23:05:33.311971 13982 raft_consensus.cc:2229] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:33.312315 13982 raft_consensus.cc:2258] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:33.315091 13983 ts_tablet_manager.cc:1905] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260805 23:05:33.316541 13726 connection.cc:535] server connection from 127.0.0.1:39352 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260805 23:05:33.317800 13983 ts_tablet_manager.cc:1918] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:33.317900 13983 log.cc:1196] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/c35f8eb504f048a1a56184b7415ec967
I20260805 23:05:33.318219 13983 ts_tablet_manager.cc:1939] T c35f8eb504f048a1a56184b7415ec967 P b40a7bc2fdef4f0f808206ad3eeec5be: Deleting consensus metadata
W20260805 23:05:33.319291 13525 connection.cc:441] server connection from 127.0.0.1:37352 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37352 (request call id 1) could send its response
I20260805 23:05:33.320551 13982 ts_tablet_manager.cc:1905] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:33.322993 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.298845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.3.12.125:34147)
0805 23:05:33.298944 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.298949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.308752 (+ 9803us) server_negotiation.cc:407] Connection header received
0805 23:05:33.308959 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.308965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.309030 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.309106 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.310448 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.311349 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.313477 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.313858 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.313973 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.316085 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.316092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.316098 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.316133 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.320792 (+ 4659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.321446 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.321450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.321452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.321546 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.321884 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.321890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.321895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.322153 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.322264 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.322664 (+ 400us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.322797 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260805 23:05:33.323160 13982 ts_tablet_manager.cc:1918] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:33.323256 13982 log.cc:1196] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/c35f8eb504f048a1a56184b7415ec967
I20260805 23:05:33.323565 13982 ts_tablet_manager.cc:1939] T c35f8eb504f048a1a56184b7415ec967 P a616b04439d245d4b24b4b0f3c164363: Deleting consensus metadata
W20260805 23:05:33.324746 13327 connection.cc:441] server connection from 127.0.0.1:59882 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59882 (request call id 1) could send its response
I20260805 23:05:33.332029 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.320807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.3.12.125:34147)
0805 23:05:33.321080 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.321086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.321746 (+ 660us) server_negotiation.cc:407] Connection header received
0805 23:05:33.321914 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.321916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.321963 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.322012 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.323535 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.324325 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.325979 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.326303 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.326377 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.328100 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.328107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.328113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.328147 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.330125 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.330678 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.330682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.330684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.330769 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.331069 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.331072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.331074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.331275 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.331377 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.331733 (+ 356us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.331849 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260805 23:05:33.333736 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.298608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.3.12.124:42281)
0805 23:05:33.298723 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.298728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.323841 (+ 25113us) server_negotiation.cc:407] Connection header received
0805 23:05:33.323997 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.324000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.324059 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.324109 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.325100 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.325850 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.327217 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.327619 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.327711 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.329499 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.329503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.329505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.329533 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.331414 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.331965 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.331971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.331973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.332066 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.332780 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.332786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.332790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.333001 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.333116 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.333455 (+ 339us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.333585 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260805 23:05:33.334448 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:33.320956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.3.12.124:42281)
0805 23:05:33.321074 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:33.321079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:33.325073 (+ 3994us) server_negotiation.cc:407] Connection header received
0805 23:05:33.325500 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:33.325504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:33.325549 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:33.325611 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:33.326524 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.327305 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.328502 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:33.328899 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:33.329020 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:33.330775 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:33.330780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:33.330786 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:33.330814 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:33.332630 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.333110 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.333113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.333115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.333196 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:33.333495 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:33.333498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:33.333500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:33.333678 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:33.333780 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:33.334182 (+ 402us) server_negotiation.cc:299] Negotiation successful
0805 23:05:33.334311 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260805 23:05:34.283004 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.282330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:38629)
0805 23:05:34.282749 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.282866 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":115,"threads_started":1}
I20260805 23:05:34.283308 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.282815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:45033)
0805 23:05:34.283111 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.283206 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":68,"threads_started":1}
W20260805 23:05:34.283484 13689 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:44519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:34.283519 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.283075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.65:59221)
0805 23:05:34.283332 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.283421 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260805 23:05:34.283556 13895 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:44519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
W20260805 23:05:34.284014 13492 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:44519 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:34.796070 14003 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47cc7c4c2a414f83804aa6688a6f6a9e)
I20260805 23:05:34.796043 14002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47cc7c4c2a414f83804aa6688a6f6a9e)
I20260805 23:05:34.796329 14002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } }
I20260805 23:05:34.796331 14003 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } }
I20260805 23:05:34.798051 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.797602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60630)
0805 23:05:34.797856 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.797938 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
I20260805 23:05:34.798198 14003 leader_election.cc:288] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519), 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124:42281)
W20260805 23:05:34.798239 13204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:34.798491 14002 leader_election.cc:288] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519), eee475496b804e56ac72865c49d6fad0 (127.3.12.125:34147)
I20260805 23:05:34.798602 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.797612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60638)
0805 23:05:34.798421 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.798496 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":63,"threads_started":1}
W20260805 23:05:34.798771 12943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:34.799371 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.799059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60652)
0805 23:05:34.799217 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.799282 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":16}
W20260805 23:05:34.799547 13204 leader_election.cc:334] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519): Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:34.799718 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.799448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60666)
0805 23:05:34.799600 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.799648 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260805 23:05:34.799854 12943 leader_election.cc:334] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519): Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:34.803196 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.797706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42281 (local address 127.0.0.1:36452)
0805 23:05:34.798742 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.798759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:34.798903 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:34.799309 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:34.799315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:34.799336 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:34.799448 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:34.799455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:34.800541 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:34.800548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:34.801850 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:34.801861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:34.802752 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:34.802756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:34.802855 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:34.802881 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 23:05:34.802964 (+ 83us) client_negotiation.cc:240] Negotiation successful
0805 23:05:34.803038 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":44,"threads_started":1}
I20260805 23:05:34.803226 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.798029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.3.12.124:42281)
0805 23:05:34.798992 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:34.798999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:34.799014 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:34.799057 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:34.799060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:34.799104 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:34.799190 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:34.799565 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:34.800412 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:34.801991 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:34.802633 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:34.802738 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:34.802801 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:34.802998 (+ 197us) server_negotiation.cc:299] Negotiation successful
0805 23:05:34.803077 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":93,"threads_started":1}
I20260805 23:05:34.803228 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.798036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34147 (local address 127.0.0.1:46174)
0805 23:05:34.798693 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.798708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:34.798804 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:34.799402 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:34.799404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:34.799420 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:34.799505 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:34.799515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:34.800585 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:34.800588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:34.801850 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:34.801861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:34.802752 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:34.802756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:34.802855 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:34.802881 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 23:05:34.802964 (+ 83us) client_negotiation.cc:240] Negotiation successful
0805 23:05:34.803038 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":96,"threads_started":1}
I20260805 23:05:34.803228 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.798339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.3.12.125:34147)
0805 23:05:34.799185 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:34.799189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:34.799201 (+ 12us) server_negotiation.cc:407] Connection header received
0805 23:05:34.799240 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:34.799243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:34.799286 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:34.799611 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:34.799693 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:34.800459 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:34.801991 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:34.802622 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:34.802738 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:34.802801 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:34.802998 (+ 197us) server_negotiation.cc:299] Negotiation successful
0805 23:05:34.803097 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":63,"threads_started":1}
I20260805 23:05:34.803825 12976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "504c0d936fa440e8bab92a2cee3d7c40" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eee475496b804e56ac72865c49d6fad0" is_pre_election: true
I20260805 23:05:34.803907 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee475496b804e56ac72865c49d6fad0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "504c0d936fa440e8bab92a2cee3d7c40" is_pre_election: true
I20260805 23:05:34.804037 12976 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504c0d936fa440e8bab92a2cee3d7c40 in term 1.
I20260805 23:05:34.804183 13233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee475496b804e56ac72865c49d6fad0 in term 1.
I20260805 23:05:34.804476 13200 leader_election.cc:302] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 504c0d936fa440e8bab92a2cee3d7c40, eee475496b804e56ac72865c49d6fad0; no voters: 47cc7c4c2a414f83804aa6688a6f6a9e
I20260805 23:05:34.804561 12940 leader_election.cc:302] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 504c0d936fa440e8bab92a2cee3d7c40, eee475496b804e56ac72865c49d6fad0; no voters: 47cc7c4c2a414f83804aa6688a6f6a9e
I20260805 23:05:34.804759 14003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 23:05:34.804816 14003 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47cc7c4c2a414f83804aa6688a6f6a9e)
I20260805 23:05:34.804855 14003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 1 FOLLOWER]: Advancing to term 2
I20260805 23:05:34.804870 14002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 23:05:34.804926 14002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47cc7c4c2a414f83804aa6688a6f6a9e)
I20260805 23:05:34.804975 14002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 1 FOLLOWER]: Advancing to term 2
I20260805 23:05:34.805884 14003 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } }
I20260805 23:05:34.806147 14002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } }
I20260805 23:05:34.806241 14003 leader_election.cc:288] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 2 election: Requested vote from peers 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519), 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124:42281)
I20260805 23:05:34.806542 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee475496b804e56ac72865c49d6fad0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "504c0d936fa440e8bab92a2cee3d7c40"
I20260805 23:05:34.806691 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.806361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60680)
0805 23:05:34.806512 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.806568 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260805 23:05:34.806734 13233 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eee475496b804e56ac72865c49d6fad0 in current term 2: Already voted for candidate 504c0d936fa440e8bab92a2cee3d7c40 in this term.
I20260805 23:05:34.806540 14002 leader_election.cc:288] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [CANDIDATE]: Term 2 election: Requested vote from peers 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519), eee475496b804e56ac72865c49d6fad0 (127.3.12.125:34147)
I20260805 23:05:34.806947 12976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "504c0d936fa440e8bab92a2cee3d7c40" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eee475496b804e56ac72865c49d6fad0"
I20260805 23:05:34.807117 12976 raft_consensus.cc:2379] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 504c0d936fa440e8bab92a2cee3d7c40 in current term 2: Already voted for candidate eee475496b804e56ac72865c49d6fad0 in this term.
I20260805 23:05:34.807403 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.807200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60700)
0805 23:05:34.807284 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.807335 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260805 23:05:34.807457 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.806681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60692)
0805 23:05:34.807341 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.807395 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565}
W20260805 23:05:34.807542 12943 leader_election.cc:334] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519): Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:34.807614 12943 leader_election.cc:302] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eee475496b804e56ac72865c49d6fad0; no voters: 47cc7c4c2a414f83804aa6688a6f6a9e, 504c0d936fa440e8bab92a2cee3d7c40
I20260805 23:05:34.807749 14003 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 23:05:34.808058 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:34.807859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:60708)
0805 23:05:34.807956 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:34.808001 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260805 23:05:34.808171 13204 leader_election.cc:334] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519): Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:34.808236 13204 leader_election.cc:302] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 504c0d936fa440e8bab92a2cee3d7c40; no voters: 47cc7c4c2a414f83804aa6688a6f6a9e, eee475496b804e56ac72865c49d6fad0
I20260805 23:05:34.808423 14002 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 23:05:35.284930 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:35.284397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:41969)
0805 23:05:35.284737 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:35.284808 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":89,"threads_started":1}
I20260805 23:05:35.285182 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:35.284630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.65:45835)
0805 23:05:35.284920 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:35.284979 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":71,"threads_started":1}
I20260805 23:05:35.285763 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:35.284060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:38861)
0805 23:05:35.285578 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:35.285665 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1430,"thread_start_us":126,"threads_started":1}
I20260805 23:05:36.286381 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.285878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:33251)
0805 23:05:36.286158 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.286237 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260805 23:05:36.286785 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.285957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.65:37631)
0805 23:05:36.286606 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.286677 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":80,"threads_started":1}
I20260805 23:05:36.286952 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.286462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:50361)
0805 23:05:36.286784 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.286850 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":84,"threads_started":1}
I20260805 23:05:36.691596 14024 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:36.691747 14024 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } }
I20260805 23:05:36.692335 14024 leader_election.cc:288] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519), 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124:42281)
I20260805 23:05:36.692834 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee475496b804e56ac72865c49d6fad0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "504c0d936fa440e8bab92a2cee3d7c40" is_pre_election: true
I20260805 23:05:36.692937 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.692566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:56188)
0805 23:05:36.692794 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.692850 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260805 23:05:36.693001 13233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eee475496b804e56ac72865c49d6fad0 in term 2.
I20260805 23:05:36.693334 12940 leader_election.cc:302] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 504c0d936fa440e8bab92a2cee3d7c40, eee475496b804e56ac72865c49d6fad0; no voters:
I20260805 23:05:36.693611 14024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 23:05:36.693717 14024 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:36.693794 14024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 2 FOLLOWER]: Advancing to term 3
I20260805 23:05:36.693813 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.693566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:56198)
0805 23:05:36.693679 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.693738 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260805 23:05:36.693931 12943 leader_election.cc:334] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519): Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:36.694877 14024 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } }
I20260805 23:05:36.695220 14024 leader_election.cc:288] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 3 election: Requested vote from peers 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519), 504c0d936fa440e8bab92a2cee3d7c40 (127.3.12.124:42281)
I20260805 23:05:36.695525 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.695306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:56206)
0805 23:05:36.695391 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.695442 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260805 23:05:36.695634 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eee475496b804e56ac72865c49d6fad0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "504c0d936fa440e8bab92a2cee3d7c40"
I20260805 23:05:36.695765 13233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 2 FOLLOWER]: Advancing to term 3
I20260805 23:05:36.696194 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.695955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:56214)
0805 23:05:36.696086 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.696139 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260805 23:05:36.696321 12943 leader_election.cc:334] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519): Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
I20260805 23:05:36.696964 13233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eee475496b804e56ac72865c49d6fad0 in term 3.
I20260805 23:05:36.697275 12940 leader_election.cc:302] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 504c0d936fa440e8bab92a2cee3d7c40, eee475496b804e56ac72865c49d6fad0; no voters: 47cc7c4c2a414f83804aa6688a6f6a9e
I20260805 23:05:36.697459 14024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 23:05:36.697655 14024 raft_consensus.cc:686] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [term 3 LEADER]: Becoming Leader. State: Replica: eee475496b804e56ac72865c49d6fad0, State: Running, Role: LEADER
I20260805 23:05:36.697917 14024 consensus_queue.cc:237] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } }
I20260805 23:05:36.698975 14029 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee475496b804e56ac72865c49d6fad0. Latest consensus state: current_term: 3 leader_uuid: "eee475496b804e56ac72865c49d6fad0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:36.699100 14029 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:36.699400 14031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:36.701786 14031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:36.702195 14031 catalog_manager.cc:1259] Loaded cluster ID: 7281e640834947a28ebb27d1e58020cd
I20260805 23:05:36.702241 14031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:36.702802 14031 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:36.703203 14031 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Loaded TSK: 0
I20260805 23:05:36.703893 14031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:05:36.717825 12964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:36.721182 12964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:36.724051 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.723812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:56222)
0805 23:05:36.723930 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.723986 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260805 23:05:36.724162 13233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [term 3 FOLLOWER]: Refusing update from remote peer eee475496b804e56ac72865c49d6fad0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260805 23:05:36.724678 14029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260805 23:05:36.724936 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.724693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.0.0.1:56232)
0805 23:05:36.724796 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.724876 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260805 23:05:36.725114 12943 consensus_peers.cc:487] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 -> Peer 47cc7c4c2a414f83804aa6688a6f6a9e (127.3.12.126:44519): Couldn't send request to peer 47cc7c4c2a414f83804aa6688a6f6a9e. Status: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 23:05:36.726166 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee475496b804e56ac72865c49d6fad0. Latest consensus state: current_term: 3 leader_uuid: "eee475496b804e56ac72865c49d6fad0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:36.726307 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:05:36.727344 14029 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "eee475496b804e56ac72865c49d6fad0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:36.727454 14029 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:36.727641 14034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "eee475496b804e56ac72865c49d6fad0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47cc7c4c2a414f83804aa6688a6f6a9e" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 44519 } } peers { permanent_uuid: "eee475496b804e56ac72865c49d6fad0" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34147 } attrs { promote: false } } peers { permanent_uuid: "504c0d936fa440e8bab92a2cee3d7c40" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42281 } attrs { promote: false } } }
I20260805 23:05:36.727792 14034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504c0d936fa440e8bab92a2cee3d7c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:05:36.738137 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.733312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:40099 (local address 127.0.0.1:44508)
0805 23:05:36.733411 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.733427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:36.733524 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:36.734472 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:36.734476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:36.734495 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:36.734609 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.734615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.735571 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.735584 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.736684 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.736718 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.737720 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.737724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.737817 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.737840 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:05:36.737911 (+ 71us) client_negotiation.cc:240] Negotiation successful
0805 23:05:36.737982 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260805 23:05:36.738371 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.733907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:46457 (local address 127.0.0.1:38302)
0805 23:05:36.734375 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.734390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:36.734474 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:36.734927 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:36.734930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:36.734947 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:36.735045 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.735050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.736065 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.736068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.737229 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.737238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.737943 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.737947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.738040 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.738078 (+ 38us) client_negotiation.cc:769] Sending connection context
0805 23:05:36.738168 (+ 90us) client_negotiation.cc:240] Negotiation successful
0805 23:05:36.738237 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":42,"thread_start_us":82,"threads_started":1}
I20260805 23:05:36.738379 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.734020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.3.12.67:46457)
0805 23:05:36.734696 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:36.734699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:36.734709 (+ 10us) server_negotiation.cc:407] Connection header received
0805 23:05:36.734779 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:36.734783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:36.734829 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:36.734922 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:36.735168 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.735924 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.737352 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.737779 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.737929 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.737995 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:36.738188 (+ 193us) server_negotiation.cc:299] Negotiation successful
0805 23:05:36.738264 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":89,"threads_started":1}
I20260805 23:05:36.738705 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.733749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:37743 (local address 127.0.0.1:35064)
0805 23:05:36.734152 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.734166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:36.734260 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:36.734802 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:36.734805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:36.734822 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:36.734935 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.734941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.736013 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.736016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.737238 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.737248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.738225 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.738229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.738330 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.738358 (+ 28us) client_negotiation.cc:769] Sending connection context
0805 23:05:36.738435 (+ 77us) client_negotiation.cc:240] Negotiation successful
0805 23:05:36.738582 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":26,"thread_start_us":90,"threads_started":1}
I20260805 23:05:36.739059 13796 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec2f2efb56f49d8914adfffba5b10bb (DEFAULT_TABLE table=test-table [id=3671b08bb07e47859d43281e03224503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:36.739125 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.733423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.3.12.66:40099)
0805 23:05:36.734226 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:36.734232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:36.734247 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:36.734305 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:36.734309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:36.734362 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:36.734462 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:36.734753 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.735457 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.736855 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.737483 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.737594 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.737660 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:36.738908 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0805 23:05:36.738996 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":76,"threads_started":1}
I20260805 23:05:36.739158 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.733866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.3.12.65:37743)
0805 23:05:36.734529 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:36.734534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:36.734548 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:36.734621 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:36.734624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:36.734687 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:36.734774 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:36.735068 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.735889 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.737381 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.737991 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.738100 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.738161 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:36.738937 (+ 776us) server_negotiation.cc:299] Negotiation successful
0805 23:05:36.739017 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":93,"threads_started":1}
I20260805 23:05:36.739356 13796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec2f2efb56f49d8914adfffba5b10bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:36.739631 13389 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec2f2efb56f49d8914adfffba5b10bb (DEFAULT_TABLE table=test-table [id=3671b08bb07e47859d43281e03224503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:36.739648 13595 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec2f2efb56f49d8914adfffba5b10bb (DEFAULT_TABLE table=test-table [id=3671b08bb07e47859d43281e03224503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:36.739935 13595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec2f2efb56f49d8914adfffba5b10bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:36.739935 13389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec2f2efb56f49d8914adfffba5b10bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:36.742070 14052 tablet_bootstrap.cc:492] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: Bootstrap starting.
I20260805 23:05:36.742728 14054 tablet_bootstrap.cc:492] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: Bootstrap starting.
I20260805 23:05:36.742972 14053 tablet_bootstrap.cc:492] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363: Bootstrap starting.
I20260805 23:05:36.743137 14052 tablet_bootstrap.cc:654] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:36.743814 14054 tablet_bootstrap.cc:654] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:36.743929 14053 tablet_bootstrap.cc:654] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:36.744602 14052 tablet_bootstrap.cc:492] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: No bootstrap required, opened a new log
I20260805 23:05:36.744699 14052 ts_tablet_manager.cc:1397] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:36.745168 14052 raft_consensus.cc:348] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.745303 14052 raft_consensus.cc:374] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:36.745350 14052 raft_consensus.cc:729] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f097662686240abbf18cd043df2a7f5, State: Initialized, Role: FOLLOWER
I20260805 23:05:36.745483 14053 tablet_bootstrap.cc:492] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363: No bootstrap required, opened a new log
I20260805 23:05:36.745493 14052 consensus_queue.cc:260] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.745576 14053 ts_tablet_manager.cc:1397] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:36.745858 14052 ts_tablet_manager.cc:1428] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:36.746022 14053 raft_consensus.cc:348] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.746142 14053 raft_consensus.cc:374] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:36.746176 14053 raft_consensus.cc:729] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a616b04439d245d4b24b4b0f3c164363, State: Initialized, Role: FOLLOWER
I20260805 23:05:36.746591 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.746323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:58217)
0805 23:05:36.746438 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.746500 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260805 23:05:36.746670 14053 consensus_queue.cc:260] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.747056 14053 ts_tablet_manager.cc:1428] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260805 23:05:36.747536 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.747284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:53237)
0805 23:05:36.747424 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.747473 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260805 23:05:36.748395 14054 tablet_bootstrap.cc:492] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: No bootstrap required, opened a new log
I20260805 23:05:36.748473 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.747977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.65:32923)
0805 23:05:36.748334 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.748390 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260805 23:05:36.748502 14054 ts_tablet_manager.cc:1397] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260805 23:05:36.748864 13896 heartbeater.cc:503] Master 127.3.12.125:34147 was elected leader, sending a full tablet report...
I20260805 23:05:36.748898 13493 heartbeater.cc:503] Master 127.3.12.125:34147 was elected leader, sending a full tablet report...
I20260805 23:05:36.748930 14054 raft_consensus.cc:348] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.749035 14054 raft_consensus.cc:374] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:36.749068 14054 raft_consensus.cc:729] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40a7bc2fdef4f0f808206ad3eeec5be, State: Initialized, Role: FOLLOWER
I20260805 23:05:36.749205 14054 consensus_queue.cc:260] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.749588 14054 ts_tablet_manager.cc:1428] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260805 23:05:36.750948 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.750665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:50391)
0805 23:05:36.750814 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.750874 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260805 23:05:36.751868 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.751599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:52481)
0805 23:05:36.751748 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.751798 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260805 23:05:36.752110 13691 heartbeater.cc:503] Master 127.3.12.125:34147 was elected leader, sending a full tablet report...
I20260805 23:05:36.765038 14063 raft_consensus.cc:482] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:36.765184 14063 raft_consensus.cc:504] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.766232 14063 leader_election.cc:288] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40a7bc2fdef4f0f808206ad3eeec5be (127.3.12.66:40099), a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743)
I20260805 23:05:36.769970 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.766495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:40099 (local address 127.3.12.67:38947)
0805 23:05:36.766649 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.766667 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:36.766771 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:36.767031 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:36.767035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:36.767063 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:36.767192 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.767197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.767937 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.767941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.768847 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.768857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.769524 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.769528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.769640 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.769659 (+ 19us) client_negotiation.cc:769] Sending connection context
0805 23:05:36.769745 (+ 86us) client_negotiation.cc:240] Negotiation successful
0805 23:05:36.769834 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":2,"spinlock_wait_cycles":2304}
I20260805 23:05:36.769968 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.766637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:38947 (local address 127.3.12.66:40099)
0805 23:05:36.766759 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:36.766764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:36.766779 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:36.766870 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:36.766874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:36.766929 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:36.767017 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:36.767306 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.767820 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.768977 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.769376 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.769498 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.769580 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:36.769777 (+ 197us) server_negotiation.cc:299] Negotiation successful
0805 23:05:36.769849 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:05:36.770124 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.766650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:50035 (local address 127.3.12.65:37743)
0805 23:05:36.766807 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:36.766815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:36.767099 (+ 284us) server_negotiation.cc:407] Connection header received
0805 23:05:36.767260 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:36.767263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:36.767308 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:36.767402 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:36.767685 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.768173 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.769177 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.769567 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.769690 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.769748 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:36.769933 (+ 185us) server_negotiation.cc:299] Negotiation successful
0805 23:05:36.770006 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260805 23:05:36.770179 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.766495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:37743 (local address 127.3.12.67:50035)
0805 23:05:36.766977 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.766989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:36.767129 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:36.767424 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:36.767426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:36.767442 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:36.767561 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.767567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.768310 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.768315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.769045 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:36.769054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.769694 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.769698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:36.769799 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.769827 (+ 28us) client_negotiation.cc:769] Sending connection context
0805 23:05:36.769917 (+ 90us) client_negotiation.cc:240] Negotiation successful
0805 23:05:36.769988 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":148,"spinlock_wait_cycles":1280,"thread_start_us":135,"threads_started":1}
I20260805 23:05:36.770663 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec2f2efb56f49d8914adfffba5b10bb" candidate_uuid: "6f097662686240abbf18cd043df2a7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a616b04439d245d4b24b4b0f3c164363" is_pre_election: true
I20260805 23:05:36.770798 13422 raft_consensus.cc:2454] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f097662686240abbf18cd043df2a7f5 in term 0.
I20260805 23:05:36.770735 13620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec2f2efb56f49d8914adfffba5b10bb" candidate_uuid: "6f097662686240abbf18cd043df2a7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" is_pre_election: true
I20260805 23:05:36.771035 13620 raft_consensus.cc:2454] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f097662686240abbf18cd043df2a7f5 in term 0.
I20260805 23:05:36.771283 13732 leader_election.cc:302] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f097662686240abbf18cd043df2a7f5, a616b04439d245d4b24b4b0f3c164363; no voters:
I20260805 23:05:36.771598 14063 raft_consensus.cc:2781] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:36.771675 14063 raft_consensus.cc:482] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:36.771741 14063 raft_consensus.cc:3037] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:36.773180 14063 raft_consensus.cc:504] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.773541 14063 leader_election.cc:288] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [CANDIDATE]: Term 1 election: Requested vote from peers b40a7bc2fdef4f0f808206ad3eeec5be (127.3.12.66:40099), a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743)
I20260805 23:05:36.773883 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec2f2efb56f49d8914adfffba5b10bb" candidate_uuid: "6f097662686240abbf18cd043df2a7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a616b04439d245d4b24b4b0f3c164363"
I20260805 23:05:36.774019 13422 raft_consensus.cc:3037] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:36.774045 13620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec2f2efb56f49d8914adfffba5b10bb" candidate_uuid: "6f097662686240abbf18cd043df2a7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be"
I20260805 23:05:36.774173 13620 raft_consensus.cc:3037] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:36.774994 13422 raft_consensus.cc:2454] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f097662686240abbf18cd043df2a7f5 in term 1.
I20260805 23:05:36.775167 13620 raft_consensus.cc:2454] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f097662686240abbf18cd043df2a7f5 in term 1.
I20260805 23:05:36.775388 13732 leader_election.cc:302] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f097662686240abbf18cd043df2a7f5, a616b04439d245d4b24b4b0f3c164363; no voters:
I20260805 23:05:36.775599 14063 raft_consensus.cc:2781] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:36.775799 14063 raft_consensus.cc:686] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 1 LEADER]: Becoming Leader. State: Replica: 6f097662686240abbf18cd043df2a7f5, State: Running, Role: LEADER
I20260805 23:05:36.776060 14063 consensus_queue.cc:237] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } }
I20260805 23:05:36.776623 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.776373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:54819)
0805 23:05:36.776472 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.776524 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260805 23:05:36.778575 12963 catalog_manager.cc:5495] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f097662686240abbf18cd043df2a7f5 (127.3.12.67). New cstate: current_term: 1 leader_uuid: "6f097662686240abbf18cd043df2a7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f097662686240abbf18cd043df2a7f5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46457 } health_report { overall_health: HEALTHY } } }
I20260805 23:05:36.791105 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.787871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.3.12.67:46457)
0805 23:05:36.788034 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:36.788038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:36.788051 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:05:36.788117 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:36.788118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:36.788170 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:36.788329 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:36.788540 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.789032 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.789844 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:36.790048 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:36.790136 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:36.790358 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:36.790785 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:36.790874 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:36.790938 (+ 64us) server_negotiation.cc:299] Negotiation successful
0805 23:05:36.790985 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260805 23:05:36.793963 13422 raft_consensus.cc:1261] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 1 FOLLOWER]: Refusing update from remote peer 6f097662686240abbf18cd043df2a7f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:36.794175 13620 raft_consensus.cc:1261] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 FOLLOWER]: Refusing update from remote peer 6f097662686240abbf18cd043df2a7f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:36.794538 14063 consensus_queue.cc:1035] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a616b04439d245d4b24b4b0f3c164363" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 37743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:36.794777 14075 consensus_queue.cc:1035] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40a7bc2fdef4f0f808206ad3eeec5be" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 40099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:36.796657 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.796368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:48121)
0805 23:05:36.796520 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.796578 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260805 23:05:36.798046 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.797225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.65:50435)
0805 23:05:36.797898 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.797959 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590}
I20260805 23:05:36.799341 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.799062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:44039)
0805 23:05:36.799211 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.799271 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260805 23:05:36.801122 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.800864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.65:47021)
0805 23:05:36.801002 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.801056 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260805 23:05:36.801962 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.801546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.67:46889)
0805 23:05:36.801849 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.801901 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260805 23:05:36.804625 14081 mvcc.cc:205] Tried to move back new op lower bound from 7315337776302809088 to 7315337776237580288. Current Snapshot: MvccSnapshot[applied={T|T < 7315337776302809088}]
I20260805 23:05:36.807548 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:36.807267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:44519 (local address 127.3.12.66:38041)
0805 23:05:36.807401 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.807465 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:44519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260805 23:05:36.807902 12966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:36.808084 12966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:36.812034 12966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eee475496b804e56ac72865c49d6fad0: Sending DeleteTablet for 3 replicas of tablet 9ec2f2efb56f49d8914adfffba5b10bb
I20260805 23:05:36.812885 13796 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec2f2efb56f49d8914adfffba5b10bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:36 UTC) from {username='oryx', principal='oryx/127.3.12.125@KRBTEST.COM'} at 127.0.0.1:38302
I20260805 23:05:36.813015 13389 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec2f2efb56f49d8914adfffba5b10bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:36 UTC) from {username='oryx', principal='oryx/127.3.12.125@KRBTEST.COM'} at 127.0.0.1:35064
I20260805 23:05:36.813357 13595 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec2f2efb56f49d8914adfffba5b10bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:36 UTC) from {username='oryx', principal='oryx/127.3.12.125@KRBTEST.COM'} at 127.0.0.1:44508
I20260805 23:05:36.813388 14102 tablet_replica.cc:330] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363: stopping tablet replica
I20260805 23:05:36.813606 14102 raft_consensus.cc:2229] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:36.813714 14100 tablet_replica.cc:330] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: stopping tablet replica
I20260805 23:05:36.813833 14100 raft_consensus.cc:2229] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:36.813862 14102 raft_consensus.cc:2258] T 9ec2f2efb56f49d8914adfffba5b10bb P a616b04439d245d4b24b4b0f3c164363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:36.814146 14100 raft_consensus.cc:2258] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:36.814163 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 13167
I20260805 23:05:36.818058 14103 tablet_replica.cc:330] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: stopping tablet replica
I20260805 23:05:36.818202 14103 raft_consensus.cc:2229] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:36.818322 14103 raft_consensus.cc:2258] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:36.820459 14103 ts_tablet_manager.cc:1905] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:36.823071 14103 ts_tablet_manager.cc:1918] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:36.823181 14103 log.cc:1196] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/9ec2f2efb56f49d8914adfffba5b10bb
I20260805 23:05:36.823462 14103 ts_tablet_manager.cc:1939] T 9ec2f2efb56f49d8914adfffba5b10bb P b40a7bc2fdef4f0f808206ad3eeec5be: Deleting consensus metadata
I20260805 23:05:36.824210 12940 catalog_manager.cc:4841] TS b40a7bc2fdef4f0f808206ad3eeec5be (127.3.12.66:40099): tablet 9ec2f2efb56f49d8914adfffba5b10bb (table test-table [id=3671b08bb07e47859d43281e03224503]) successfully deleted
I20260805 23:05:36.832959 14100 ts_tablet_manager.cc:1905] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260805 23:05:36.834352 14049 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:36.833691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:37743 (local address 127.0.0.1:35068)
0805 23:05:36.833886 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:36.833903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:36.833994 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:36.834233 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:37743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":106}
W20260805 23:05:36.834692 12944 catalog_manager.cc:4584] TS a616b04439d245d4b24b4b0f3c164363 (127.3.12.65:37743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ec2f2efb56f49d8914adfffba5b10bb: Network error: Client connection negotiation failed: client connection to 127.3.12.65:37743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260805 23:05:36.834908 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 13501
I20260805 23:05:36.839752 14100 ts_tablet_manager.cc:1918] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:36.839884 14100 log.cc:1196] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/9ec2f2efb56f49d8914adfffba5b10bb
I20260805 23:05:36.840356 14100 ts_tablet_manager.cc:1939] T 9ec2f2efb56f49d8914adfffba5b10bb P 6f097662686240abbf18cd043df2a7f5: Deleting consensus metadata
I20260805 23:05:36.841451 12941 catalog_manager.cc:4841] TS 6f097662686240abbf18cd043df2a7f5 (127.3.12.67:46457): tablet 9ec2f2efb56f49d8914adfffba5b10bb (table test-table [id=3671b08bb07e47859d43281e03224503]) successfully deleted
I20260805 23:05:36.853535 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 13697
I20260805 23:05:36.870143 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 12752
I20260805 23:05:36.881757 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 12913
2026-08-05T23:05:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7495 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:36 dist-test-slave-2d19 krb5kdc[14106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:36 dist-test-slave-2d19 krb5kdc[14106](info): set up 2 sockets
Aug 05 23:05:36 dist-test-slave-2d19 krb5kdc[14106](info): commencing operation
krb5kdc: starting...
W20260805 23:05:38.113106 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:38 dist-test-slave-2d19 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.126.keytab.
Entry for principal oryx/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.126.keytab.
I20260805 23:05:38.247592 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:38489
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.126.keytab
--principal=oryx/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:38.359382 14128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:38.359666 14128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:38.359709 14128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:38.363725 14128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:38.363790 14128 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260805 23:05:38.363807 14128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:38.363826 14128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:38.363842 14128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:38.363857 14128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:38.369256 14128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:38489
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.126.keytab
--principal=oryx/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:38.370481 14128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:38.371624 14128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:38.381162 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:38.381157 14136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:38.381157 14138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:38.381436 14128 server_base.cc:1034] running on GCE node
I20260805 23:05:38.382085 14128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:38.383307 14128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:38.384539 14128 hybrid_clock.cc:648] HybridClock initialized: now 1785971138384524 us; error 42 us; skew 500 ppm
Aug 05 23:05:38 dist-test-slave-2d19 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971138, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:38.394137 14128 init.cc:377] Logged in from keytab as oryx/127.3.12.126@KRBTEST.COM (short username oryx)
I20260805 23:05:38.395251 14128 webserver.cc:460] Webserver started at http://127.3.12.126:38911/ using document root <none> and password file <none>
I20260805 23:05:38.395870 14128 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:38.395968 14128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:38.396224 14128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:38.398437 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "a3a813a82c5947dbbf1be0c0a4348fe4"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "2199614389406ffa4477785f9b602496"
server_key_iv: "7b06427da3a6a5eec455168d4d461faf"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.399045 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "a3a813a82c5947dbbf1be0c0a4348fe4"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "2199614389406ffa4477785f9b602496"
server_key_iv: "7b06427da3a6a5eec455168d4d461faf"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.402961 14128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260805 23:05:38.405622 14147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:38.406883 14128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:38.407104 14128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "a3a813a82c5947dbbf1be0c0a4348fe4"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "2199614389406ffa4477785f9b602496"
server_key_iv: "7b06427da3a6a5eec455168d4d461faf"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.407225 14128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:38.429986 14128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:38.432950 14128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:38.433167 14128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:38.441454 14128 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:38489
I20260805 23:05:38.441459 14235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:38489 every 8 connection(s)
I20260805 23:05:38.442445 14128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:38.445328 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 14128
I20260805 23:05:38.445417 14237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:38.445523 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:38.445828 3121 external_mini_cluster.cc:1428] Setting key 0bb34b69a36a45d06e5d5275b14a0ebc
I20260805 23:05:38.452878 14237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: Bootstrap starting.
I20260805 23:05:38.455493 14237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: Neither blocks nor log segments found. Creating new log.
Aug 05 23:05:38 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:38.456506 14237 log.cc:824] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:38.458776 14237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: No bootstrap required, opened a new log
I20260805 23:05:38.462263 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.447386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.3.12.126:38489)
0805 23:05:38.447854 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:38.447875 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:38.447920 (+ 45us) server_negotiation.cc:407] Connection header received
0805 23:05:38.448970 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:38.448996 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:38.449436 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:38.449829 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:38.450573 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.452092 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.453063 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.453561 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.453755 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.456758 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:38.456764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:38.456779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:38.456808 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:38.459278 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.459811 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.459816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.459821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.459929 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.460226 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.460229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.460230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.460681 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:38.460822 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:38.461163 (+ 341us) server_negotiation.cc:299] Negotiation successful
0805 23:05:38.461475 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":123,"threads_started":1}
I20260805 23:05:38.462520 14237 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a813a82c5947dbbf1be0c0a4348fe4" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38489 } }
I20260805 23:05:38.462785 14237 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:38.462881 14237 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a813a82c5947dbbf1be0c0a4348fe4, State: Initialized, Role: FOLLOWER
I20260805 23:05:38.463415 14237 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a813a82c5947dbbf1be0c0a4348fe4" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38489 } }
I20260805 23:05:38.463586 14237 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:38.463654 14237 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:38.463753 14237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:38.464943 14237 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a813a82c5947dbbf1be0c0a4348fe4" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38489 } }
I20260805 23:05:38.465245 14237 leader_election.cc:302] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3a813a82c5947dbbf1be0c0a4348fe4; no voters:
I20260805 23:05:38.465495 14237 leader_election.cc:288] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:38.465674 14247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:38.465919 14247 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [term 1 LEADER]: Becoming Leader. State: Replica: a3a813a82c5947dbbf1be0c0a4348fe4, State: Running, Role: LEADER
I20260805 23:05:38.466284 14247 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a813a82c5947dbbf1be0c0a4348fe4" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38489 } }
I20260805 23:05:38.466789 14237 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:38.468037 14249 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3a813a82c5947dbbf1be0c0a4348fe4. Latest consensus state: current_term: 1 leader_uuid: "a3a813a82c5947dbbf1be0c0a4348fe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a813a82c5947dbbf1be0c0a4348fe4" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38489 } } }
I20260805 23:05:38.468174 14249 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:38.468400 14248 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3a813a82c5947dbbf1be0c0a4348fe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a813a82c5947dbbf1be0c0a4348fe4" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 38489 } } }
I20260805 23:05:38.468488 14248 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:38.468972 14263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:38.471942 14263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:38.477624 14263 catalog_manager.cc:1347] Generated new cluster ID: 0a13025513354b80bbccec8f5de9c021
I20260805 23:05:38.477705 14263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:38.491729 14263 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:38.492826 14263 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:38.498862 14263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: Generated new TSK 0
I20260805 23:05:38.499580 14263 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab.
Entry for principal oryx/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab.
I20260805 23:05:38.536949 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38489
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab
--principal=oryx/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:38.655072 14285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:38.655362 14285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:38.655409 14285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:38.659754 14285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:38.659838 14285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:38.659881 14285 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:38.659958 14285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:38.665567 14285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab
--principal=oryx/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:38.666693 14285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:38.667896 14285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:05:38.677883 14285 server_base.cc:1034] running on GCE node
W20260805 23:05:38.677901 14295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:38.677927 14293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:38.678102 14292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:38.678553 14285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:38.679179 14285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:38.680331 14285 hybrid_clock.cc:648] HybridClock initialized: now 1785971138680298 us; error 46 us; skew 500 ppm
Aug 05 23:05:38 dist-test-slave-2d19 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971138, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:38.683131 14285 init.cc:377] Logged in from keytab as oryx/127.3.12.65@KRBTEST.COM (short username oryx)
I20260805 23:05:38.684015 14285 webserver.cc:460] Webserver started at http://127.3.12.65:33131/ using document root <none> and password file <none>
I20260805 23:05:38.684747 14285 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:38.684842 14285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:38.685103 14285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:38.687263 14285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "086cb35b832146f9b01b50dbd9cf4b0d"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "7d1149e9207a1373781fdd321e2d2599"
server_key_iv: "b3c34d6f6f135af908894878f725eec4"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.687788 14285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "086cb35b832146f9b01b50dbd9cf4b0d"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "7d1149e9207a1373781fdd321e2d2599"
server_key_iv: "b3c34d6f6f135af908894878f725eec4"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.691699 14285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:05:38.694653 14307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:38.695914 14285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:38.696039 14285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "086cb35b832146f9b01b50dbd9cf4b0d"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "7d1149e9207a1373781fdd321e2d2599"
server_key_iv: "b3c34d6f6f135af908894878f725eec4"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.696175 14285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:38.713204 14285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:38.716609 14285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:38.716823 14285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:38.718339 14285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:38.718389 14285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:38.718431 14285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:38.718480 14285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:38 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971138, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:38.729528 14285 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:42977
I20260805 23:05:38.729910 14467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:42977 every 8 connection(s)
I20260805 23:05:38.730506 14285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:38.732795 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.719143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:39691 (local address 127.3.12.126:38489)
0805 23:05:38.719299 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:38.719304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:38.720206 (+ 902us) server_negotiation.cc:407] Connection header received
0805 23:05:38.721254 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:38.721260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:38.721325 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:38.721390 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:38.723248 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.724172 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.725664 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.725923 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.726046 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.728739 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:38.728745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:38.728753 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:38.728781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:38.730467 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.731102 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.731105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.731107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.731203 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.731617 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.731621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.731623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.731840 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:38.731946 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:38.732556 (+ 610us) server_negotiation.cc:299] Negotiation successful
0805 23:05:38.732671 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260805 23:05:38.733669 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.719570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38489 (local address 127.3.12.65:39691)
0805 23:05:38.720024 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:38.720075 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:38.720981 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:38.721875 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:38.721883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:38.722394 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:38.723063 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:38.723084 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.724345 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.724350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:38.725526 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:38.725537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.726104 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.726108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:38.726280 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.726879 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:38.726895 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:38.728559 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:38.730597 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:38.730606 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:38.730624 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:38.730946 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:38.731315 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:38.731318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:38.731320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:38.731504 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:38.731995 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:38.732001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:38.732237 (+ 236us) client_negotiation.cc:769] Sending connection context
0805 23:05:38.732506 (+ 269us) client_negotiation.cc:240] Negotiation successful
0805 23:05:38.732771 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":134,"threads_started":1}
I20260805 23:05:38.735167 14468 heartbeater.cc:347] Connected to a master server at 127.3.12.126:38489
I20260805 23:05:38.735191 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 14285
I20260805 23:05:38.735320 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:38.735339 14468 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:38.735694 3121 external_mini_cluster.cc:1428] Setting key 573b63c30a5039595235f71834070fb3
I20260805 23:05:38.735926 14468 heartbeater.cc:511] Master 127.3.12.126:38489 requested a full tablet report, sending...
I20260805 23:05:38.737596 14169 ts_manager.cc:194] Registered new tserver with Master: 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977)
I20260805 23:05:38.738749 14169 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:39691
WARNING: no policy specified for oryx/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:38.747179 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.737422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38489 (local address 127.3.12.65:39647)
0805 23:05:38.737743 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:38.737761 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:38.737859 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:38.740061 (+ 2202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:38.740064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:38.740330 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:38.740526 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:38.740536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.741560 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.741562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:38.742105 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:38.742111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.742586 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.742589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:38.742681 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.743279 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:38.743291 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:38.743649 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:38.745558 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:38.745561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:38.745563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:38.745775 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:38.746075 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:38.746078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:38.746080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:38.746166 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:38.746596 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:38.746600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:38.746699 (+ 99us) client_negotiation.cc:769] Sending connection context
0805 23:05:38.746809 (+ 110us) client_negotiation.cc:240] Negotiation successful
0805 23:05:38.746979 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":16,"thread_start_us":112,"threads_started":1}
I20260805 23:05:38.747161 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.739621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:39647 (local address 127.3.12.126:38489)
0805 23:05:38.739759 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:38.739763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:38.739778 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:38.739815 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:38.739818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:38.739863 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:38.739947 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:38.740675 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.741396 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.742223 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.742461 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.742577 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.743786 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:38.743789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:38.743792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:38.743816 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:38.745411 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.745882 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.745886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.745888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.745975 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.746259 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.746263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.746265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.746498 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:38.746607 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:38.746866 (+ 259us) server_negotiation.cc:299] Negotiation successful
0805 23:05:38.746978 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab.
Entry for principal oryx/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab.
I20260805 23:05:38.768309 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38489
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab
--principal=oryx/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:38.884122 14482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:38.884460 14482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:38.884565 14482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:38.888705 14482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:38.888763 14482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:38.888803 14482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:38.888864 14482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:38.894198 14482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab
--principal=oryx/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:38.895421 14482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:38.896560 14482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:38.906443 14492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:38.906452 14491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:38.906454 14495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:38.906817 14482 server_base.cc:1034] running on GCE node
I20260805 23:05:38.907253 14482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:38.908003 14482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:38.909206 14482 hybrid_clock.cc:648] HybridClock initialized: now 1785971138909190 us; error 37 us; skew 500 ppm
Aug 05 23:05:38 dist-test-slave-2d19 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971138, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:38.912544 14482 init.cc:377] Logged in from keytab as oryx/127.3.12.66@KRBTEST.COM (short username oryx)
I20260805 23:05:38.913424 14482 webserver.cc:460] Webserver started at http://127.3.12.66:40357/ using document root <none> and password file <none>
I20260805 23:05:38.913992 14482 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:38.914060 14482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:38.914238 14482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:38.916566 14482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "7b6ef6eed92842c3a4cf0e11a98660ba"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "4c51365c3d6149fd7841ea52b987704c"
server_key_iv: "a5790630953a5ee50697ad8b17886fe2"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.917215 14482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "7b6ef6eed92842c3a4cf0e11a98660ba"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "4c51365c3d6149fd7841ea52b987704c"
server_key_iv: "a5790630953a5ee50697ad8b17886fe2"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.921134 14482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:38.923537 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:38.924880 14482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:38.925014 14482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "7b6ef6eed92842c3a4cf0e11a98660ba"
format_stamp: "Formatted at 2026-08-05 23:05:38 on dist-test-slave-2d19"
server_key: "4c51365c3d6149fd7841ea52b987704c"
server_key_iv: "a5790630953a5ee50697ad8b17886fe2"
server_key_version: "encryptionkey@0"
I20260805 23:05:38.925119 14482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:38.946729 14482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:38.950254 14482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:38.950489 14482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:38.952046 14482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:38.952099 14482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:38.952143 14482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:38.952200 14482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:38 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971138, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:38.963797 14482 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:41305
I20260805 23:05:38.963874 14675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:41305 every 8 connection(s)
I20260805 23:05:38.964936 14482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:38.966457 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 14482
I20260805 23:05:38.966598 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:38.966938 3121 external_mini_cluster.cc:1428] Setting key 667b1c76174b63d7526bc07893ad5a66
I20260805 23:05:38.966948 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.953039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:46431 (local address 127.3.12.126:38489)
0805 23:05:38.953203 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:38.953208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:38.954135 (+ 927us) server_negotiation.cc:407] Connection header received
0805 23:05:38.955287 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:38.955311 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:38.955382 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:38.955470 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:38.957436 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.958126 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.959273 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.959556 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.959676 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.962753 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:38.962756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:38.962758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:38.962784 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:38.964474 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.965078 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.965082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.965084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.965174 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.965610 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.965615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.965619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.965856 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:38.965947 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:38.966607 (+ 660us) server_negotiation.cc:299] Negotiation successful
0805 23:05:38.966756 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 23:05:38.967841 14522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.953428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38489 (local address 127.3.12.66:46431)
0805 23:05:38.953945 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:38.953997 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:38.955001 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:38.955975 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:38.955984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:38.956490 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:38.957212 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:38.957229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.958266 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.958268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:38.959139 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:38.959148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.959738 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.959742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:38.959925 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.960599 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:38.960616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:38.962563 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:38.964591 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:38.964597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:38.964611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:38.964965 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:38.965278 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:38.965282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:38.965283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:38.965479 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:38.965963 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:38.965969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:38.966263 (+ 294us) client_negotiation.cc:769] Sending connection context
0805 23:05:38.966487 (+ 224us) client_negotiation.cc:240] Negotiation successful
0805 23:05:38.966835 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":181,"threads_started":1}
I20260805 23:05:38.968974 14676 heartbeater.cc:347] Connected to a master server at 127.3.12.126:38489
I20260805 23:05:38.969163 14676 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:38.969738 14676 heartbeater.cc:511] Master 127.3.12.126:38489 requested a full tablet report, sending...
I20260805 23:05:38.970906 14170 ts_manager.cc:194] Registered new tserver with Master: 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305)
I20260805 23:05:38.971720 14170 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:46431
WARNING: no policy specified for oryx/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:05:38.979079 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.971675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:45849 (local address 127.3.12.126:38489)
0805 23:05:38.971800 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:38.971805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:38.971820 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:38.971862 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:38.971866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:38.971933 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:38.972044 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:38.972639 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.973209 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.974286 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.974510 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.974625 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.975753 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:38.975757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:38.975760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:38.975784 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:38.977299 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.977762 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.977765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.977767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.977852 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:38.978166 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:38.978171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:38.978172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:38.978382 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:38.978505 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:38.978805 (+ 300us) server_negotiation.cc:299] Negotiation successful
0805 23:05:38.978926 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260805 23:05:38.979094 14686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:38.971287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38489 (local address 127.3.12.66:45849)
0805 23:05:38.971593 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:38.971608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:38.971699 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:38.972097 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:38.972099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:38.972360 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:38.972504 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:38.972512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.973351 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.973354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:38.974156 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:38.974166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:38.974640 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:38.974643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:38.974740 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:38.975284 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:38.975296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:38.975610 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:38.977443 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:38.977446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:38.977448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:38.977655 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:38.977977 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:38.977980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:38.977982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:38.978071 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:38.978506 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:38.978509 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:38.978613 (+ 104us) client_negotiation.cc:769] Sending connection context
0805 23:05:38.978745 (+ 132us) client_negotiation.cc:240] Negotiation successful
0805 23:05:38.978902 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":3,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab.
Entry for principal oryx/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab.
I20260805 23:05:38.999959 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:38489
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab
--principal=oryx/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:39.122258 14689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:39.122563 14689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:39.122614 14689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:39.127188 14689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:39.127255 14689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:39.127300 14689 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260805 23:05:39.127337 14689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:39.133672 14689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab
--principal=oryx/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.12.126:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:39.134835 14689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:39.135980 14689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:39.146364 14699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:39.146397 14696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:39.146435 14697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:39.146621 14689 server_base.cc:1034] running on GCE node
I20260805 23:05:39.147113 14689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:39.147728 14689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:39.148897 14689 hybrid_clock.cc:648] HybridClock initialized: now 1785971139148876 us; error 30 us; skew 500 ppm
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971139, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:39.151813 14689 init.cc:377] Logged in from keytab as oryx/127.3.12.67@KRBTEST.COM (short username oryx)
I20260805 23:05:39.152801 14689 webserver.cc:460] Webserver started at http://127.3.12.67:41727/ using document root <none> and password file <none>
I20260805 23:05:39.153379 14689 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:39.153496 14689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:39.153751 14689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:39.155793 14689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "7acc628c5e8c469ca391c52857e972fe"
format_stamp: "Formatted at 2026-08-05 23:05:39 on dist-test-slave-2d19"
server_key: "e5545f1b819dd831d49cd79f0a5afb95"
server_key_iv: "7fed083e8385c337f0629b218601ad71"
server_key_version: "encryptionkey@0"
I20260805 23:05:39.156438 14689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "7acc628c5e8c469ca391c52857e972fe"
format_stamp: "Formatted at 2026-08-05 23:05:39 on dist-test-slave-2d19"
server_key: "e5545f1b819dd831d49cd79f0a5afb95"
server_key_iv: "7fed083e8385c337f0629b218601ad71"
server_key_version: "encryptionkey@0"
I20260805 23:05:39.160401 14689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:39.162978 14709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:39.164187 14689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:39.164374 14689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "7acc628c5e8c469ca391c52857e972fe"
format_stamp: "Formatted at 2026-08-05 23:05:39 on dist-test-slave-2d19"
server_key: "e5545f1b819dd831d49cd79f0a5afb95"
server_key_iv: "7fed083e8385c337f0629b218601ad71"
server_key_version: "encryptionkey@0"
I20260805 23:05:39.164515 14689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:39.188884 14689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:39.192003 14689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:39.192221 14689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:39.193825 14689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:39.193903 14689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:39.193975 14689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:39.194029 14689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971139, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:39.205564 14689 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:41893
I20260805 23:05:39.205641 14877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:41893 every 8 connection(s)
I20260805 23:05:39.206764 14689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:39.207903 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 14689
I20260805 23:05:39.208053 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:39.208433 3121 external_mini_cluster.cc:1428] Setting key cf7e7531abb7f21bfeb6fdb52070d1bf
I20260805 23:05:39.209821 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.194729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:39925 (local address 127.3.12.126:38489)
0805 23:05:39.194859 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.194863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.196049 (+ 1186us) server_negotiation.cc:407] Connection header received
0805 23:05:39.197155 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.197158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.197219 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.197331 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:39.199147 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.200095 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.202081 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.202382 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.202471 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.205735 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:39.205739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:39.205742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:39.205770 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:39.207447 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.208047 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.208051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.208054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.208154 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.208531 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.208534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.208536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.208731 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:39.208857 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.209526 (+ 669us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.209658 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:05:39.210848 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.195403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38489 (local address 127.3.12.67:39925)
0805 23:05:39.195894 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.195936 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.196922 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.197653 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.197666 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.198220 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:39.198954 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.198979 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.200225 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.200232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.201925 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.201941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.202548 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.202552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.202778 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.203497 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:39.203526 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:39.205496 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:39.207559 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:39.207567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:39.207583 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:39.207926 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:39.208236 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:39.208240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:39.208242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:39.208418 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:39.208919 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:39.208927 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:39.209234 (+ 307us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.209451 (+ 217us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.209804 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":142,"threads_started":1}
I20260805 23:05:39.212106 14878 heartbeater.cc:347] Connected to a master server at 127.3.12.126:38489
I20260805 23:05:39.212268 14878 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:39.212826 14878 heartbeater.cc:511] Master 127.3.12.126:38489 requested a full tablet report, sending...
I20260805 23:05:39.213907 14169 ts_manager.cc:194] Registered new tserver with Master: 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:39.214738 14169 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:39925
I20260805 23:05:39.221029 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.214379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:46363 (local address 127.3.12.126:38489)
0805 23:05:39.214486 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.214490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.214700 (+ 210us) server_negotiation.cc:407] Connection header received
0805 23:05:39.214807 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.214809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.214876 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.214986 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:39.215502 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.216069 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.216944 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.217163 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.217274 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.218266 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:39.218270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:39.218272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:39.218302 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:39.219375 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.219797 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.219800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.219802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.219881 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.220180 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.220185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.220186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.220410 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:39.220526 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.220784 (+ 258us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.220892 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260805 23:05:39.221045 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.214309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38489 (local address 127.3.12.67:46363)
0805 23:05:39.214618 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.214632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.214727 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.215028 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.215031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.215246 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:39.215376 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.215383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.216193 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.216197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.216823 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.216832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.217280 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.217283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.217374 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.217803 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:39.217814 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:39.218129 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:39.219488 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:39.219492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:39.219494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:39.219690 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:39.219976 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:39.219979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:39.219981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:39.220087 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:39.220510 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:39.220513 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:39.220604 (+ 91us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.220716 (+ 112us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.220862 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260805 23:05:39.222723 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260805 23:05:39.251374 14245 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:39.244861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.3.12.126:38489)
0805 23:05:39.245006 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.245009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.245023 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:39.245085 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.245088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.245128 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.245207 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:39.245952 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.246477 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.247322 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.247522 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.247598 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.251157 (+ 3559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39942: BlockingRecv error: recv got EOF from 127.0.0.1:39942 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:39.261179 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.253167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.3.12.126:38489)
0805 23:05:39.253317 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.253323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.253399 (+ 76us) server_negotiation.cc:407] Connection header received
0805 23:05:39.253573 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.253576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.253626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.253711 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:39.254307 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.254820 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.255641 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.255872 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.255989 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.258343 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:39.258346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:39.258349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:39.258372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:39.259630 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.260014 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.260017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.260019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.260093 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.260471 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.260474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.260476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.260657 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:39.260770 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.260968 (+ 198us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.261060 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260805 23:05:39.263952 14169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:39.267123 14169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:39.281857 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.276394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:41893 (local address 127.0.0.1:52098)
0805 23:05:39.276680 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.276693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.276772 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.277795 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.277798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.277818 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:39.277931 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.277938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.279040 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.279043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.280406 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.280419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.281313 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.281316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.281419 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.281458 (+ 39us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.281588 (+ 130us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.281711 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260805 23:05:39.281868 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.276830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.3.12.67:41893)
0805 23:05:39.277318 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.277334 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.277358 (+ 24us) server_negotiation.cc:407] Connection header received
0805 23:05:39.277442 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.277447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.277647 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.277825 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:39.278066 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.278915 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.280521 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.281188 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.281311 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.281488 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.281619 (+ 131us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.281712 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":90,"threads_started":1}
I20260805 23:05:39.282163 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.276733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42977 (local address 127.0.0.1:45118)
0805 23:05:39.276946 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.276958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.277039 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.278042 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.278045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.278064 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:39.278167 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.278173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.279524 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.279528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.280747 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.280758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.281776 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.281780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.281890 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.281912 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.281965 (+ 53us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.282045 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":5,"thread_start_us":98,"threads_started":1}
I20260805 23:05:39.282800 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.275953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:41305 (local address 127.0.0.1:48146)
0805 23:05:39.276475 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.276529 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.276673 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.277536 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.277542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.277662 (+ 120us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:39.277814 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.277821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.279475 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.279479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.280675 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.280683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.281776 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.281780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.282503 (+ 723us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.282526 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.282619 (+ 93us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.282682 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":678,"thread_start_us":71,"threads_started":1}
I20260805 23:05:39.282843 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.276099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.3.12.66:41305)
0805 23:05:39.277001 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.277017 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.277040 (+ 23us) server_negotiation.cc:407] Connection header received
0805 23:05:39.277123 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.277131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.277352 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.277971 (+ 619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:39.278119 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.279310 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.280793 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.281624 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.281828 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.282009 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.282652 (+ 643us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.282732 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":192,"threads_started":1}
I20260805 23:05:39.283504 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.277262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.3.12.65:42977)
0805 23:05:39.277626 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.277634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.277657 (+ 23us) server_negotiation.cc:407] Connection header received
0805 23:05:39.277729 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.277734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.277911 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.278042 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:39.278285 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.279345 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.280854 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.281624 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.283082 (+ 1458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.283226 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.283276 (+ 50us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.283367 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":71,"threads_started":1}
I20260805 23:05:39.285346 14780 tablet_service.cc:1467] Processing CreateTablet for tablet fdf605fd78d04d55a5ed0f68984b787e (DEFAULT_TABLE table=test-table [id=a428e363ee874811b1ad283a997d2779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.285343 14576 tablet_service.cc:1467] Processing CreateTablet for tablet fdf605fd78d04d55a5ed0f68984b787e (DEFAULT_TABLE table=test-table [id=a428e363ee874811b1ad283a997d2779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.286316 14576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf605fd78d04d55a5ed0f68984b787e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.286325 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf605fd78d04d55a5ed0f68984b787e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.292900 14916 tablet_bootstrap.cc:492] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Bootstrap starting.
I20260805 23:05:39.294350 14917 tablet_bootstrap.cc:492] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Bootstrap starting.
I20260805 23:05:39.294592 14372 tablet_service.cc:1467] Processing CreateTablet for tablet fdf605fd78d04d55a5ed0f68984b787e (DEFAULT_TABLE table=test-table [id=a428e363ee874811b1ad283a997d2779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.295333 14916 tablet_bootstrap.cc:654] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.295614 14372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf605fd78d04d55a5ed0f68984b787e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.296221 14916 log.cc:824] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:39.296464 14917 tablet_bootstrap.cc:654] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.297219 14917 log.cc:824] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:39.298488 14916 tablet_bootstrap.cc:492] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: No bootstrap required, opened a new log
I20260805 23:05:39.298681 14916 ts_tablet_manager.cc:1397] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:05:39.299443 14917 tablet_bootstrap.cc:492] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: No bootstrap required, opened a new log
I20260805 23:05:39.299639 14917 ts_tablet_manager.cc:1397] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 23:05:39.302205 14922 tablet_bootstrap.cc:492] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Bootstrap starting.
I20260805 23:05:39.302651 14916 raft_consensus.cc:348] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.302925 14916 raft_consensus.cc:374] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.302990 14916 raft_consensus.cc:729] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6ef6eed92842c3a4cf0e11a98660ba, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.303493 14916 consensus_queue.cc:260] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.303499 14917 raft_consensus.cc:348] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.303740 14917 raft_consensus.cc:374] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.303810 14917 raft_consensus.cc:729] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acc628c5e8c469ca391c52857e972fe, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.304243 14917 consensus_queue.cc:260] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.304598 14922 tablet_bootstrap.cc:654] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.304605 14916 ts_tablet_manager.cc:1428] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 23:05:39.304766 14676 heartbeater.cc:503] Master 127.3.12.126:38489 was elected leader, sending a full tablet report...
I20260805 23:05:39.305138 14878 heartbeater.cc:503] Master 127.3.12.126:38489 was elected leader, sending a full tablet report...
I20260805 23:05:39.305434 14922 log.cc:824] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:39.306118 14917 ts_tablet_manager.cc:1428] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260805 23:05:39.307565 14922 tablet_bootstrap.cc:492] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: No bootstrap required, opened a new log
I20260805 23:05:39.307780 14922 ts_tablet_manager.cc:1397] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:05:39.311654 14922 raft_consensus.cc:348] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.311875 14922 raft_consensus.cc:374] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.311949 14922 raft_consensus.cc:729] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cb35b832146f9b01b50dbd9cf4b0d, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.312399 14922 consensus_queue.cc:260] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.312979 14468 heartbeater.cc:503] Master 127.3.12.126:38489 was elected leader, sending a full tablet report...
I20260805 23:05:39.313184 14922 ts_tablet_manager.cc:1428] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260805 23:05:39.457813 14879 tablet.cc:2369] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:05:39.457960 14924 raft_consensus.cc:482] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:39.458128 14924 raft_consensus.cc:504] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.459583 14924 leader_election.cc:288] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305), 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977)
I20260805 23:05:39.464465 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.459930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:41305 (local address 127.3.12.67:41113)
0805 23:05:39.460079 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.460095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.460199 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.460615 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.460618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.460642 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:39.460791 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.460801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.461909 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.461913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.463131 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.463141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.464001 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.464004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.464117 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.464144 (+ 27us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.464234 (+ 90us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.464331 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":1}
I20260805 23:05:39.464465 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.460035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:41113 (local address 127.3.12.66:41305)
0805 23:05:39.460216 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.460222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.460300 (+ 78us) server_negotiation.cc:407] Connection header received
0805 23:05:39.460373 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.460375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.460423 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.460514 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:39.460941 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.461787 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.463291 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.463901 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.464001 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.464066 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.464274 (+ 208us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.464354 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 23:05:39.465134 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.460226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42977 (local address 127.3.12.67:52195)
0805 23:05:39.460670 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.460690 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.460776 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.461037 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.461040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.461065 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:39.461597 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.461606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.462683 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.462685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.463831 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.463843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.464689 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.464694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.464805 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.464835 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.464907 (+ 72us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.464958 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":26,"thread_start_us":176,"threads_started":1}
I20260805 23:05:39.465143 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.460448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:52195 (local address 127.3.12.65:42977)
0805 23:05:39.460615 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.460619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.460779 (+ 160us) server_negotiation.cc:407] Connection header received
0805 23:05:39.460860 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.460865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.460938 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.460997 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:39.461759 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.462561 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.463969 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.464477 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.464607 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.464687 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.464944 (+ 257us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.465033 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 23:05:39.465345 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf605fd78d04d55a5ed0f68984b787e" candidate_uuid: "7acc628c5e8c469ca391c52857e972fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" is_pre_election: true
I20260805 23:05:39.465732 14608 raft_consensus.cc:2454] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7acc628c5e8c469ca391c52857e972fe in term 0.
I20260805 23:05:39.465905 14401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf605fd78d04d55a5ed0f68984b787e" candidate_uuid: "7acc628c5e8c469ca391c52857e972fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" is_pre_election: true
W20260805 23:05:39.466177 14677 tablet.cc:2369] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:05:39.466207 14401 raft_consensus.cc:2454] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7acc628c5e8c469ca391c52857e972fe in term 0.
I20260805 23:05:39.466248 14710 leader_election.cc:302] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7acc628c5e8c469ca391c52857e972fe, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:39.466537 14924 raft_consensus.cc:2781] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:39.466635 14924 raft_consensus.cc:482] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:39.466688 14924 raft_consensus.cc:3037] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.467772 14924 raft_consensus.cc:504] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.468474 14924 leader_election.cc:288] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [CANDIDATE]: Term 1 election: Requested vote from peers 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305), 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977)
I20260805 23:05:39.468528 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf605fd78d04d55a5ed0f68984b787e" candidate_uuid: "7acc628c5e8c469ca391c52857e972fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba"
I20260805 23:05:39.468679 14608 raft_consensus.cc:3037] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.469017 14401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf605fd78d04d55a5ed0f68984b787e" candidate_uuid: "7acc628c5e8c469ca391c52857e972fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cb35b832146f9b01b50dbd9cf4b0d"
I20260805 23:05:39.469161 14401 raft_consensus.cc:3037] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.469955 14608 raft_consensus.cc:2454] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7acc628c5e8c469ca391c52857e972fe in term 1.
I20260805 23:05:39.470270 14710 leader_election.cc:302] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7acc628c5e8c469ca391c52857e972fe, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:39.470481 14924 raft_consensus.cc:2781] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:39.470570 14401 raft_consensus.cc:2454] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7acc628c5e8c469ca391c52857e972fe in term 1.
I20260805 23:05:39.470746 14924 raft_consensus.cc:686] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 1 LEADER]: Becoming Leader. State: Replica: 7acc628c5e8c469ca391c52857e972fe, State: Running, Role: LEADER
I20260805 23:05:39.471120 14924 consensus_queue.cc:237] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.474560 14170 catalog_manager.cc:5495] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7acc628c5e8c469ca391c52857e972fe (127.3.12.67). New cstate: current_term: 1 leader_uuid: "7acc628c5e8c469ca391c52857e972fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } health_report { overall_health: HEALTHY } } }
W20260805 23:05:39.481978 14469 tablet.cc:2369] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:05:39.518002 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.514360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.3.12.67:41893)
0805 23:05:39.514504 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.514508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.514525 (+ 17us) server_negotiation.cc:407] Connection header received
0805 23:05:39.514627 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.514630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.514686 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.514801 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:39.515066 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.515564 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.516498 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.516719 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.516834 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.517131 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:39.517589 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:39.517727 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.517788 (+ 61us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.517851 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:05:39.524150 14608 raft_consensus.cc:1261] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Refusing update from remote peer 7acc628c5e8c469ca391c52857e972fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:39.524147 14401 raft_consensus.cc:1261] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Refusing update from remote peer 7acc628c5e8c469ca391c52857e972fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:39.524940 14924 consensus_queue.cc:1035] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:39.525151 14929 consensus_queue.cc:1035] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:39.541296 14170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:39.541500 14170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:39.544025 14170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: Sending DeleteTablet for 3 replicas of tablet fdf605fd78d04d55a5ed0f68984b787e
I20260805 23:05:39.544883 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf605fd78d04d55a5ed0f68984b787e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:39 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:52098
I20260805 23:05:39.544922 14372 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf605fd78d04d55a5ed0f68984b787e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:39 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:45118
I20260805 23:05:39.545455 14949 tablet_replica.cc:330] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: stopping tablet replica
I20260805 23:05:39.545436 14576 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf605fd78d04d55a5ed0f68984b787e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:39 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:48146
I20260805 23:05:39.545883 14952 tablet_replica.cc:330] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: stopping tablet replica
I20260805 23:05:39.545913 14949 raft_consensus.cc:2229] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:39.546197 14952 raft_consensus.cc:2229] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:39.546231 14170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260805 23:05:39.546505 14952 raft_consensus.cc:2258] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Raft consensus is shut down!
W20260805 23:05:39.546650 14170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:39.546662 14949 raft_consensus.cc:2258] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:39.547268 14947 tablet_replica.cc:330] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: stopping tablet replica
I20260805 23:05:39.547610 14947 raft_consensus.cc:2229] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:39.548060 14947 raft_consensus.cc:2258] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:39.548128 14952 ts_tablet_manager.cc:1905] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:39.549696 14947 ts_tablet_manager.cc:1905] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:39.551128 14952 ts_tablet_manager.cc:1918] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:39.551254 14952 log.cc:1196] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/fdf605fd78d04d55a5ed0f68984b787e
I20260805 23:05:39.551635 14952 ts_tablet_manager.cc:1939] T fdf605fd78d04d55a5ed0f68984b787e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting consensus metadata
I20260805 23:05:39.552842 14947 ts_tablet_manager.cc:1918] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:39.552943 14947 log.cc:1196] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/fdf605fd78d04d55a5ed0f68984b787e
I20260805 23:05:39.553046 14372 tablet_service.cc:1467] Processing CreateTablet for tablet b84ae95e6a3547e99b6631ed549b01d9 (DEFAULT_TABLE table=test-table [id=94421f5144034406bedf1cccf15f3269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.553251 14947 ts_tablet_manager.cc:1939] T fdf605fd78d04d55a5ed0f68984b787e P 7acc628c5e8c469ca391c52857e972fe: Deleting consensus metadata
I20260805 23:05:39.553371 14372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84ae95e6a3547e99b6631ed549b01d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.555253 14922 tablet_bootstrap.cc:492] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: Bootstrap starting.
I20260805 23:05:39.555469 14576 tablet_service.cc:1467] Processing CreateTablet for tablet b84ae95e6a3547e99b6631ed549b01d9 (DEFAULT_TABLE table=test-table [id=94421f5144034406bedf1cccf15f3269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.555764 14576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84ae95e6a3547e99b6631ed549b01d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.556294 14922 tablet_bootstrap.cc:654] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.556784 14148 catalog_manager.cc:4841] TS 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305): tablet fdf605fd78d04d55a5ed0f68984b787e (table test-table [id=a428e363ee874811b1ad283a997d2779]) successfully deleted
I20260805 23:05:39.558055 14916 tablet_bootstrap.cc:492] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: Bootstrap starting.
I20260805 23:05:39.558099 14149 catalog_manager.cc:4841] TS 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893): tablet fdf605fd78d04d55a5ed0f68984b787e (table test-table [id=a428e363ee874811b1ad283a997d2779]) successfully deleted
I20260805 23:05:39.558446 14780 tablet_service.cc:1467] Processing CreateTablet for tablet b84ae95e6a3547e99b6631ed549b01d9 (DEFAULT_TABLE table=test-table [id=94421f5144034406bedf1cccf15f3269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.558805 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84ae95e6a3547e99b6631ed549b01d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.558990 14916 tablet_bootstrap.cc:654] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.559190 14949 ts_tablet_manager.cc:1905] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:39.561271 14917 tablet_bootstrap.cc:492] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: Bootstrap starting.
I20260805 23:05:39.562003 14949 ts_tablet_manager.cc:1918] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:39.562172 14949 log.cc:1196] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/fdf605fd78d04d55a5ed0f68984b787e
I20260805 23:05:39.562230 14916 tablet_bootstrap.cc:492] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: No bootstrap required, opened a new log
I20260805 23:05:39.562237 14917 tablet_bootstrap.cc:654] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.562325 14916 ts_tablet_manager.cc:1397] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260805 23:05:39.562877 14916 raft_consensus.cc:348] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.563024 14916 raft_consensus.cc:374] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.563084 14916 raft_consensus.cc:729] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6ef6eed92842c3a4cf0e11a98660ba, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.563262 14949 ts_tablet_manager.cc:1939] T fdf605fd78d04d55a5ed0f68984b787e P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting consensus metadata
I20260805 23:05:39.563278 14916 consensus_queue.cc:260] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.563984 14916 ts_tablet_manager.cc:1428] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:39.564776 14148 catalog_manager.cc:4841] TS 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977): tablet fdf605fd78d04d55a5ed0f68984b787e (table test-table [id=a428e363ee874811b1ad283a997d2779]) successfully deleted
I20260805 23:05:39.564864 14922 tablet_bootstrap.cc:492] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: No bootstrap required, opened a new log
I20260805 23:05:39.564944 14922 ts_tablet_manager.cc:1397] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260805 23:05:39.565399 14922 raft_consensus.cc:348] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.565531 14922 raft_consensus.cc:374] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.565577 14922 raft_consensus.cc:729] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cb35b832146f9b01b50dbd9cf4b0d, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.565716 14922 consensus_queue.cc:260] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.566095 14922 ts_tablet_manager.cc:1428] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:39.567762 14917 tablet_bootstrap.cc:492] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: No bootstrap required, opened a new log
I20260805 23:05:39.567883 14917 ts_tablet_manager.cc:1397] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260805 23:05:39.568449 14917 raft_consensus.cc:348] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.568604 14917 raft_consensus.cc:374] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.568646 14917 raft_consensus.cc:729] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acc628c5e8c469ca391c52857e972fe, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.568787 14917 consensus_queue.cc:260] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.569078 14917 ts_tablet_manager.cc:1428] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260805 23:05:39.576092 14926 raft_consensus.cc:482] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:39.576229 14926 raft_consensus.cc:504] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.577366 14926 leader_election.cc:288] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305), 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:39.582113 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.577532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:41305 (local address 127.3.12.65:39195)
0805 23:05:39.577941 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.577955 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.578112 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.578740 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.578742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.578763 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:39.578900 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.578908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.579774 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.579780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.580810 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.580821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.581634 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.581638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.581753 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.581780 (+ 27us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.581866 (+ 86us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.581937 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":56,"spinlock_wait_cycles":5632,"thread_start_us":101,"threads_started":1}
I20260805 23:05:39.582155 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.577657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:39195 (local address 127.3.12.66:41305)
0805 23:05:39.577860 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.577868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.578032 (+ 164us) server_negotiation.cc:407] Connection header received
0805 23:05:39.578263 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.578266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.578315 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.578366 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:39.579081 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.579626 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.580940 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.581498 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.581613 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.581677 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.581923 (+ 246us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.582002 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260805 23:05:39.582698 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.578016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:55733 (local address 127.3.12.67:41893)
0805 23:05:39.578129 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.578133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.578173 (+ 40us) server_negotiation.cc:407] Connection header received
0805 23:05:39.578223 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.578226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.578268 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.578332 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:39.578739 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.579509 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.581240 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.581809 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.581956 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.582013 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.582494 (+ 481us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.582566 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260805 23:05:39.582727 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84ae95e6a3547e99b6631ed549b01d9" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" is_pre_election: true
I20260805 23:05:39.582870 14608 raft_consensus.cc:2454] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 0.
I20260805 23:05:39.583180 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.577616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:41893 (local address 127.3.12.65:55733)
0805 23:05:39.577974 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.577989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.578112 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.578449 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.578453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.578472 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:39.578612 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.578618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.579653 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.579657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.581106 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.581118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.582093 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.582097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.582327 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.582365 (+ 38us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.582860 (+ 495us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.582953 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":77,"spinlock_wait_cycles":25728,"thread_start_us":138,"threads_started":1}
I20260805 23:05:39.583305 14309 leader_election.cc:302] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cb35b832146f9b01b50dbd9cf4b0d, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:39.583545 14926 raft_consensus.cc:2781] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:39.583618 14926 raft_consensus.cc:482] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:39.583666 14926 raft_consensus.cc:3037] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.585000 14926 raft_consensus.cc:504] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.585409 14926 leader_election.cc:288] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 election: Requested vote from peers 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305), 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:39.585743 14807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84ae95e6a3547e99b6631ed549b01d9" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acc628c5e8c469ca391c52857e972fe" is_pre_election: true
I20260805 23:05:39.585831 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84ae95e6a3547e99b6631ed549b01d9" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba"
I20260805 23:05:39.585955 14608 raft_consensus.cc:3037] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.586035 14807 raft_consensus.cc:2454] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 0.
I20260805 23:05:39.585790 14806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84ae95e6a3547e99b6631ed549b01d9" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acc628c5e8c469ca391c52857e972fe"
I20260805 23:05:39.586318 14806 raft_consensus.cc:3037] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.587142 14608 raft_consensus.cc:2454] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 1.
I20260805 23:05:39.587334 14806 raft_consensus.cc:2454] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 1.
I20260805 23:05:39.587486 14309 leader_election.cc:302] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cb35b832146f9b01b50dbd9cf4b0d, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:39.587670 14926 raft_consensus.cc:2781] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:39.587901 14926 raft_consensus.cc:686] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 LEADER]: Becoming Leader. State: Replica: 086cb35b832146f9b01b50dbd9cf4b0d, State: Running, Role: LEADER
I20260805 23:05:39.588174 14926 consensus_queue.cc:237] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.590463 14168 catalog_manager.cc:5495] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65). New cstate: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } health_report { overall_health: UNKNOWN } } }
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971138, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.126@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:39.627153 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.618102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:38489 (local address 127.0.0.1:39952)
0805 23:05:39.618407 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.618427 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.618580 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.618854 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.618856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.619299 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:39.619396 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.619402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.620165 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.620171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.620823 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.620832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.621301 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.621304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.621386 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.622051 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:39.622066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:39.623750 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:39.625505 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:39.625515 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:39.625522 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:39.625763 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:39.626045 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:39.626056 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:39.626058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:39.626142 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:39.626545 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:39.626553 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:39.626673 (+ 120us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.626789 (+ 116us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.626948 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
I20260805 23:05:39.627152 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.618175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.3.12.126:38489)
0805 23:05:39.618317 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.618323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.618508 (+ 185us) server_negotiation.cc:407] Connection header received
0805 23:05:39.618697 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.618700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.618747 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.618832 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:39.619521 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.620044 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.620950 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.621173 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.621283 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.623892 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:39.623896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:39.623898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:39.623928 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:39.625381 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.625867 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.625872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.625874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.625951 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.626232 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.626237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.626238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.626414 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:39.626545 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.626844 (+ 299us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.627006 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":49}
I20260805 23:05:39.632323 14168 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:39952:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260805 23:05:39.633131 14168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:39.638414 14780 tablet_service.cc:1467] Processing CreateTablet for tablet e888ef70ee5b4bb4ad6ad2afc706fb30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a37726dc02e34299af7607a878963f9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:39.638403 14372 tablet_service.cc:1467] Processing CreateTablet for tablet e888ef70ee5b4bb4ad6ad2afc706fb30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a37726dc02e34299af7607a878963f9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:39.638417 14576 tablet_service.cc:1467] Processing CreateTablet for tablet e888ef70ee5b4bb4ad6ad2afc706fb30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a37726dc02e34299af7607a878963f9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260805 23:05:39.638828 14372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e888ef70ee5b4bb4ad6ad2afc706fb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.638828 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e888ef70ee5b4bb4ad6ad2afc706fb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.638827 14576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e888ef70ee5b4bb4ad6ad2afc706fb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.641741 14916 tablet_bootstrap.cc:492] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: Bootstrap starting.
I20260805 23:05:39.641768 14922 tablet_bootstrap.cc:492] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: Bootstrap starting.
I20260805 23:05:39.641958 14917 tablet_bootstrap.cc:492] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: Bootstrap starting.
I20260805 23:05:39.642760 14922 tablet_bootstrap.cc:654] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.642760 14917 tablet_bootstrap.cc:654] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.642798 14916 tablet_bootstrap.cc:654] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.644518 14917 tablet_bootstrap.cc:492] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: No bootstrap required, opened a new log
I20260805 23:05:39.644542 14922 tablet_bootstrap.cc:492] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: No bootstrap required, opened a new log
I20260805 23:05:39.644630 14922 ts_tablet_manager.cc:1397] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:39.644632 14917 ts_tablet_manager.cc:1397] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260805 23:05:39.645009 14917 raft_consensus.cc:348] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.645133 14917 raft_consensus.cc:374] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.645069 14922 raft_consensus.cc:348] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.645175 14917 raft_consensus.cc:729] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acc628c5e8c469ca391c52857e972fe, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.645180 14922 raft_consensus.cc:374] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.645215 14922 raft_consensus.cc:729] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cb35b832146f9b01b50dbd9cf4b0d, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.645303 14917 consensus_queue.cc:260] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.645349 14922 consensus_queue.cc:260] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.645532 14943 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.645633 14922 ts_tablet_manager.cc:1428] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260805 23:05:39.645675 14943 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:39.645605 14917 ts_tablet_manager.cc:1428] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260805 23:05:39.646265 14926 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.646396 14926 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:39.647315 14916 tablet_bootstrap.cc:492] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: No bootstrap required, opened a new log
I20260805 23:05:39.647405 14916 ts_tablet_manager.cc:1397] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260805 23:05:39.647877 14916 raft_consensus.cc:348] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.647986 14916 raft_consensus.cc:374] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.648026 14916 raft_consensus.cc:729] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6ef6eed92842c3a4cf0e11a98660ba, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.648164 14916 consensus_queue.cc:260] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.648612 14916 ts_tablet_manager.cc:1428] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:39.648818 14923 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.648952 14923 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:39.802781 14926 raft_consensus.cc:482] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:39.802942 14926 raft_consensus.cc:504] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.803448 14926 leader_election.cc:288] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305), 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:39.803956 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e888ef70ee5b4bb4ad6ad2afc706fb30" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" is_pre_election: true
I20260805 23:05:39.803980 14806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e888ef70ee5b4bb4ad6ad2afc706fb30" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acc628c5e8c469ca391c52857e972fe" is_pre_election: true
I20260805 23:05:39.804122 14608 raft_consensus.cc:2454] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 0.
I20260805 23:05:39.804170 14806 raft_consensus.cc:2454] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 0.
I20260805 23:05:39.804481 14309 leader_election.cc:302] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cb35b832146f9b01b50dbd9cf4b0d, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:39.804695 14926 raft_consensus.cc:2781] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:39.804790 14926 raft_consensus.cc:482] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:39.804848 14926 raft_consensus.cc:3037] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.805846 14926 raft_consensus.cc:504] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.806190 14926 leader_election.cc:288] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 election: Requested vote from peers 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305), 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:39.806564 14806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e888ef70ee5b4bb4ad6ad2afc706fb30" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acc628c5e8c469ca391c52857e972fe"
I20260805 23:05:39.806569 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e888ef70ee5b4bb4ad6ad2afc706fb30" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba"
I20260805 23:05:39.806684 14806 raft_consensus.cc:3037] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.806684 14608 raft_consensus.cc:3037] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:39.807803 14806 raft_consensus.cc:2454] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 1.
I20260805 23:05:39.807803 14608 raft_consensus.cc:2454] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 1.
I20260805 23:05:39.808115 14309 leader_election.cc:302] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cb35b832146f9b01b50dbd9cf4b0d, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:39.808312 14926 raft_consensus.cc:2781] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:39.808441 14926 raft_consensus.cc:686] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 LEADER]: Becoming Leader. State: Replica: 086cb35b832146f9b01b50dbd9cf4b0d, State: Running, Role: LEADER
I20260805 23:05:39.808612 14926 consensus_queue.cc:237] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.809358 14961 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: TxnStatusTablet state changed. Reason: New leader 086cb35b832146f9b01b50dbd9cf4b0d. Latest consensus state: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.809482 14961 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:39.809777 14981 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260805 23:05:39.809963 14981 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260805 23:05:39.810491 14170 catalog_manager.cc:5495] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65). New cstate: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:39.845424 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.841237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.3.12.65:42977)
0805 23:05:39.841367 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.841370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.841404 (+ 34us) server_negotiation.cc:407] Connection header received
0805 23:05:39.841638 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.841640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.841698 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.841777 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:39.842064 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.842562 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.843806 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.844047 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.844129 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.844528 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:39.845005 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:39.845111 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.845228 (+ 117us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.845280 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:05:39.845459 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.841086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42977 (local address 127.0.0.1:45120)
0805 23:05:39.841339 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.841355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.841474 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.841821 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.841824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.841835 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:39.841955 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.841961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.842672 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.842675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.843659 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.843670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.844160 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.844164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.844287 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.844347 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:39.845122 (+ 775us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:39.845138 (+ 16us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.845219 (+ 81us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.845312 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260805 23:05:39.848950 14608 raft_consensus.cc:1261] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Refusing update from remote peer 086cb35b832146f9b01b50dbd9cf4b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:39.849056 14806 raft_consensus.cc:1261] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Refusing update from remote peer 086cb35b832146f9b01b50dbd9cf4b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:39.849488 14961 consensus_queue.cc:1035] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:39.849671 14926 consensus_queue.cc:1035] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:39.851358 14923 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: TxnStatusTablet state changed. Reason: New leader 086cb35b832146f9b01b50dbd9cf4b0d. Latest consensus state: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.851481 14923 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:39.851392 14943 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: TxnStatusTablet state changed. Reason: New leader 086cb35b832146f9b01b50dbd9cf4b0d. Latest consensus state: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.851523 14943 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:39.854111 14926 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.854229 14926 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:39.854676 14933 mvcc.cc:205] Tried to move back new op lower bound from 7315337788815405056 to 7315337788657692672. Current Snapshot: MvccSnapshot[applied={T|T < 7315337788815405056}]
I20260805 23:05:39.854841 14934 mvcc.cc:205] Tried to move back new op lower bound from 7315337788815405056 to 7315337788657692672. Current Snapshot: MvccSnapshot[applied={T|T < 7315337788815405056 or (T in {7315337788815405056})}]
I20260805 23:05:39.855952 14923 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.856063 14923 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7b6ef6eed92842c3a4cf0e11a98660ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:39.856652 14961 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.856766 14961 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 086cb35b832146f9b01b50dbd9cf4b0d: This TxnStatusTablet replica's current role is: LEADER
I20260805 23:05:39.858124 14943 tablet_replica.cc:439] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } } }
I20260805 23:05:39.858247 14943 tablet_replica.cc:442] T e888ef70ee5b4bb4ad6ad2afc706fb30 P 7acc628c5e8c469ca391c52857e972fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260805 23:05:39.858140 14932 mvcc.cc:205] Tried to move back new op lower bound from 7315337788815405056 to 7315337788657692672. Current Snapshot: MvccSnapshot[applied={T|T < 7315337788815405056}]
I20260805 23:05:39.863958 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.859309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.3.12.65:42977)
0805 23:05:39.860261 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.860266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.860299 (+ 33us) server_negotiation.cc:407] Connection header received
0805 23:05:39.860338 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.860341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.860395 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.860468 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:39.860796 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.861485 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.862710 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.863003 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.863125 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.863421 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:39.863497 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:39.863567 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.863723 (+ 156us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.863772 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872}
I20260805 23:05:39.875808 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.872059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:50851 (local address 127.3.12.65:42977)
0805 23:05:39.872199 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.872203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.872497 (+ 294us) server_negotiation.cc:407] Connection header received
0805 23:05:39.872553 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.872555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.872597 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.872672 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:39.872932 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.873409 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.874677 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.874878 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.874954 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.875298 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:39.875363 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:39.875418 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.875593 (+ 175us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.875652 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:05:39.875833 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.871993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42977 (local address 127.3.12.65:50851)
0805 23:05:39.872343 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:39.872364 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:39.872482 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:39.872708 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:39.872711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:39.872718 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:39.872850 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.872855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.873512 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.873515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.874545 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:39.874565 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.875014 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.875018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:39.875111 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.875163 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:39.875455 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:39.875464 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 23:05:39.875533 (+ 69us) client_negotiation.cc:240] Negotiation successful
0805 23:05:39.875621 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":144,"threads_started":1}
I20260805 23:05:39.883210 14608 raft_consensus.cc:1261] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Refusing update from remote peer 086cb35b832146f9b01b50dbd9cf4b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:39.883240 14806 raft_consensus.cc:1261] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Refusing update from remote peer 086cb35b832146f9b01b50dbd9cf4b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:39.883636 14996 consensus_queue.cc:1035] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:39.883877 14961 consensus_queue.cc:1035] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:39.926951 14168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:39.927115 14168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:39.929669 14168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: Sending DeleteTablet for 3 replicas of tablet b84ae95e6a3547e99b6631ed549b01d9
I20260805 23:05:39.930346 14576 tablet_service.cc:1514] Processing DeleteTablet for tablet b84ae95e6a3547e99b6631ed549b01d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:39 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:48146
I20260805 23:05:39.930336 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet b84ae95e6a3547e99b6631ed549b01d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:39 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:52098
I20260805 23:05:39.930543 14947 tablet_replica.cc:330] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: stopping tablet replica
I20260805 23:05:39.930553 14371 tablet_service.cc:1514] Processing DeleteTablet for tablet b84ae95e6a3547e99b6631ed549b01d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:39 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:45118
I20260805 23:05:39.930685 14947 raft_consensus.cc:2229] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:39.930729 14949 tablet_replica.cc:330] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: stopping tablet replica
I20260805 23:05:39.930817 14947 raft_consensus.cc:2258] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:39.930831 14949 raft_consensus.cc:2229] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:39.931010 14952 tablet_replica.cc:330] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: stopping tablet replica
I20260805 23:05:39.931113 14949 raft_consensus.cc:2258] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:39.931129 14952 raft_consensus.cc:2229] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:39.931236 14952 raft_consensus.cc:2258] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:39.931839 14949 ts_tablet_manager.cc:1905] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:39.932045 14952 ts_tablet_manager.cc:1905] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:39.934741 14949 ts_tablet_manager.cc:1918] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:39.934829 14949 log.cc:1196] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/b84ae95e6a3547e99b6631ed549b01d9
I20260805 23:05:39.934872 14952 ts_tablet_manager.cc:1918] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:39.934942 14952 log.cc:1196] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/b84ae95e6a3547e99b6631ed549b01d9
I20260805 23:05:39.935115 14949 ts_tablet_manager.cc:1939] T b84ae95e6a3547e99b6631ed549b01d9 P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting consensus metadata
I20260805 23:05:39.935245 14952 ts_tablet_manager.cc:1939] T b84ae95e6a3547e99b6631ed549b01d9 P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting consensus metadata
I20260805 23:05:39.936028 14148 catalog_manager.cc:4841] TS 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305): tablet b84ae95e6a3547e99b6631ed549b01d9 (table test-table [id=94421f5144034406bedf1cccf15f3269]) successfully deleted
I20260805 23:05:39.936383 14148 catalog_manager.cc:4841] TS 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977): tablet b84ae95e6a3547e99b6631ed549b01d9 (table test-table [id=94421f5144034406bedf1cccf15f3269]) successfully deleted
I20260805 23:05:39.938637 14947 ts_tablet_manager.cc:1905] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:39.941756 14947 ts_tablet_manager.cc:1918] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:39.941843 14947 log.cc:1196] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/b84ae95e6a3547e99b6631ed549b01d9
I20260805 23:05:39.942109 14947 ts_tablet_manager.cc:1939] T b84ae95e6a3547e99b6631ed549b01d9 P 7acc628c5e8c469ca391c52857e972fe: Deleting consensus metadata
I20260805 23:05:39.942891 14149 catalog_manager.cc:4841] TS 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893): tablet b84ae95e6a3547e99b6631ed549b01d9 (table test-table [id=94421f5144034406bedf1cccf15f3269]) successfully deleted
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260805 23:05:39.962961 14245 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:39.955089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.3.12.126:38489)
0805 23:05:39.955262 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.955266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.955467 (+ 201us) server_negotiation.cc:407] Connection header received
0805 23:05:39.955642 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.955644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.955691 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.955768 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:39.956788 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.957322 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.958576 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.958817 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.958997 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.962815 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39962: BlockingRecv error: recv got EOF from 127.0.0.1:39962 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 05 23:05:39 dist-test-slave-2d19 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM
I20260805 23:05:39.975495 14245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:39.965135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.3.12.126:38489)
0805 23:05:39.965265 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:39.965271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:39.965286 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:39.965332 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:39.965335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:39.965386 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:39.965482 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:39.966419 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.967280 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.968467 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:39.968914 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:39.969035 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:39.971948 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:39.971952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:39.971955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:39.971984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:39.973679 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.974192 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.974196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.974198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.974283 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:39.974580 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:39.974583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:39.974585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:39.974796 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:39.974922 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:39.975205 (+ 283us) server_negotiation.cc:299] Negotiation successful
0805 23:05:39.975323 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:05:39.978746 14168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:05:39.979216 14168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:39.984839 14371 tablet_service.cc:1467] Processing CreateTablet for tablet d359ee46d90f468587a3eedec9b79c6e (DEFAULT_TABLE table=test-table [id=2471ced800fb4ff6bc20fb45f39a3454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.984907 14576 tablet_service.cc:1467] Processing CreateTablet for tablet d359ee46d90f468587a3eedec9b79c6e (DEFAULT_TABLE table=test-table [id=2471ced800fb4ff6bc20fb45f39a3454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.985165 14371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d359ee46d90f468587a3eedec9b79c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.985251 14576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d359ee46d90f468587a3eedec9b79c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.987699 14916 tablet_bootstrap.cc:492] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: Bootstrap starting.
I20260805 23:05:39.987774 14922 tablet_bootstrap.cc:492] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: Bootstrap starting.
I20260805 23:05:39.988602 14916 tablet_bootstrap.cc:654] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.988637 14922 tablet_bootstrap.cc:654] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.989578 14780 tablet_service.cc:1467] Processing CreateTablet for tablet d359ee46d90f468587a3eedec9b79c6e (DEFAULT_TABLE table=test-table [id=2471ced800fb4ff6bc20fb45f39a3454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:39.989795 14916 tablet_bootstrap.cc:492] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: No bootstrap required, opened a new log
I20260805 23:05:39.989871 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d359ee46d90f468587a3eedec9b79c6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:39.989894 14916 ts_tablet_manager.cc:1397] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:39.990471 14916 raft_consensus.cc:348] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.990581 14922 tablet_bootstrap.cc:492] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: No bootstrap required, opened a new log
I20260805 23:05:39.990604 14916 raft_consensus.cc:374] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.990640 14916 raft_consensus.cc:729] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6ef6eed92842c3a4cf0e11a98660ba, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.990664 14922 ts_tablet_manager.cc:1397] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260805 23:05:39.990773 14916 consensus_queue.cc:260] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.991074 14922 raft_consensus.cc:348] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.991180 14922 raft_consensus.cc:374] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.991211 14922 raft_consensus.cc:729] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cb35b832146f9b01b50dbd9cf4b0d, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.991348 14922 consensus_queue.cc:260] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.991569 14916 ts_tablet_manager.cc:1428] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:39.991605 14922 ts_tablet_manager.cc:1428] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:39.992152 14917 tablet_bootstrap.cc:492] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: Bootstrap starting.
I20260805 23:05:39.993003 14917 tablet_bootstrap.cc:654] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:39.994412 14917 tablet_bootstrap.cc:492] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: No bootstrap required, opened a new log
I20260805 23:05:39.994515 14917 ts_tablet_manager.cc:1397] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:39.994911 14917 raft_consensus.cc:348] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.995034 14917 raft_consensus.cc:374] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:39.995085 14917 raft_consensus.cc:729] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acc628c5e8c469ca391c52857e972fe, State: Initialized, Role: FOLLOWER
I20260805 23:05:39.995221 14917 consensus_queue.cc:260] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:39.995507 14917 ts_tablet_manager.cc:1428] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:40.137028 15001 raft_consensus.cc:482] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:40.137202 15001 raft_consensus.cc:504] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.137697 15001 leader_election.cc:288] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305), 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:40.138208 14807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d359ee46d90f468587a3eedec9b79c6e" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acc628c5e8c469ca391c52857e972fe" is_pre_election: true
I20260805 23:05:40.138255 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d359ee46d90f468587a3eedec9b79c6e" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" is_pre_election: true
I20260805 23:05:40.138387 14807 raft_consensus.cc:2454] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 0.
I20260805 23:05:40.138442 14608 raft_consensus.cc:2454] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 0.
I20260805 23:05:40.138765 14310 leader_election.cc:302] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cb35b832146f9b01b50dbd9cf4b0d, 7acc628c5e8c469ca391c52857e972fe; no voters:
I20260805 23:05:40.139009 15001 raft_consensus.cc:2781] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:40.139102 15001 raft_consensus.cc:482] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:40.139158 15001 raft_consensus.cc:3037] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:40.140167 15001 raft_consensus.cc:504] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.140537 15001 leader_election.cc:288] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 election: Requested vote from peers 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305), 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:40.140928 14608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d359ee46d90f468587a3eedec9b79c6e" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba"
I20260805 23:05:40.140972 14807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d359ee46d90f468587a3eedec9b79c6e" candidate_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acc628c5e8c469ca391c52857e972fe"
I20260805 23:05:40.141068 14608 raft_consensus.cc:3037] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:40.141121 14807 raft_consensus.cc:3037] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:40.141969 14608 raft_consensus.cc:2454] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 1.
I20260805 23:05:40.142184 14807 raft_consensus.cc:2454] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 086cb35b832146f9b01b50dbd9cf4b0d in term 1.
I20260805 23:05:40.142330 14309 leader_election.cc:302] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cb35b832146f9b01b50dbd9cf4b0d, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:40.142581 15001 raft_consensus.cc:2781] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:40.142675 15001 raft_consensus.cc:686] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 LEADER]: Becoming Leader. State: Replica: 086cb35b832146f9b01b50dbd9cf4b0d, State: Running, Role: LEADER
I20260805 23:05:40.142827 15001 consensus_queue.cc:237] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.144665 14168 catalog_manager.cc:5495] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65). New cstate: current_term: 1 leader_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:40.187564 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.184331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.3.12.65:42977)
0805 23:05:40.184494 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:40.184499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:40.184517 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:05:40.184576 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:40.184579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:40.184632 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:40.184718 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:40.185058 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.185556 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.186427 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.186653 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.186770 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.187053 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:40.187129 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:40.187213 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:40.187342 (+ 129us) server_negotiation.cc:299] Negotiation successful
0805 23:05:40.187403 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 23:05:40.189724 14807 raft_consensus.cc:1261] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Refusing update from remote peer 086cb35b832146f9b01b50dbd9cf4b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:40.189762 14608 raft_consensus.cc:1261] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Refusing update from remote peer 086cb35b832146f9b01b50dbd9cf4b0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:40.190364 14996 consensus_queue.cc:1035] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:40.190529 15001 consensus_queue.cc:1035] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:40.197382 14169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:40.197551 14169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:40.199707 14169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: Sending DeleteTablet for 3 replicas of tablet d359ee46d90f468587a3eedec9b79c6e
I20260805 23:05:40.200238 14371 tablet_service.cc:1514] Processing DeleteTablet for tablet d359ee46d90f468587a3eedec9b79c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:40 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:45118
I20260805 23:05:40.200368 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet d359ee46d90f468587a3eedec9b79c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:40 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:52098
I20260805 23:05:40.200408 14576 tablet_service.cc:1514] Processing DeleteTablet for tablet d359ee46d90f468587a3eedec9b79c6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:40 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:48146
I20260805 23:05:40.200471 14949 tablet_replica.cc:330] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: stopping tablet replica
I20260805 23:05:40.200600 14952 tablet_replica.cc:330] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: stopping tablet replica
I20260805 23:05:40.200614 14949 raft_consensus.cc:2229] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:40.200727 14952 raft_consensus.cc:2229] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:40.200758 14947 tablet_replica.cc:330] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: stopping tablet replica
I20260805 23:05:40.200794 14949 raft_consensus.cc:2258] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:40.200848 14952 raft_consensus.cc:2258] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:40.200870 14947 raft_consensus.cc:2229] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:40.200980 14947 raft_consensus.cc:2258] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:40.201414 14169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260805 23:05:40.201680 14952 ts_tablet_manager.cc:1905] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260805 23:05:40.201822 14169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:05:40.202210 14947 ts_tablet_manager.cc:1905] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:40.204455 14952 ts_tablet_manager.cc:1918] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:40.204538 14952 log.cc:1196] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/d359ee46d90f468587a3eedec9b79c6e
I20260805 23:05:40.204838 14952 ts_tablet_manager.cc:1939] T d359ee46d90f468587a3eedec9b79c6e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting consensus metadata
I20260805 23:05:40.204882 14947 ts_tablet_manager.cc:1918] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:40.204953 14947 log.cc:1196] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/d359ee46d90f468587a3eedec9b79c6e
I20260805 23:05:40.205220 14947 ts_tablet_manager.cc:1939] T d359ee46d90f468587a3eedec9b79c6e P 7acc628c5e8c469ca391c52857e972fe: Deleting consensus metadata
I20260805 23:05:40.205602 14148 catalog_manager.cc:4841] TS 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305): tablet d359ee46d90f468587a3eedec9b79c6e (table test-table [id=2471ced800fb4ff6bc20fb45f39a3454]) successfully deleted
I20260805 23:05:40.205961 14149 catalog_manager.cc:4841] TS 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893): tablet d359ee46d90f468587a3eedec9b79c6e (table test-table [id=2471ced800fb4ff6bc20fb45f39a3454]) successfully deleted
I20260805 23:05:40.205973 14949 ts_tablet_manager.cc:1905] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:40.207849 14949 ts_tablet_manager.cc:1918] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:05:40.207953 14949 log.cc:1196] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/d359ee46d90f468587a3eedec9b79c6e
I20260805 23:05:40.208189 14949 ts_tablet_manager.cc:1939] T d359ee46d90f468587a3eedec9b79c6e P 086cb35b832146f9b01b50dbd9cf4b0d: Deleting consensus metadata
I20260805 23:05:40.208853 14148 catalog_manager.cc:4841] TS 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977): tablet d359ee46d90f468587a3eedec9b79c6e (table test-table [id=2471ced800fb4ff6bc20fb45f39a3454]) successfully deleted
I20260805 23:05:40.212877 14371 tablet_service.cc:1467] Processing CreateTablet for tablet ee2fd8d3dd634e299d4b215411d85c3e (DEFAULT_TABLE table=test-table [id=3e4a6578da084984b45a377be1f70a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:40.212941 14576 tablet_service.cc:1467] Processing CreateTablet for tablet ee2fd8d3dd634e299d4b215411d85c3e (DEFAULT_TABLE table=test-table [id=3e4a6578da084984b45a377be1f70a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:40.213068 14780 tablet_service.cc:1467] Processing CreateTablet for tablet ee2fd8d3dd634e299d4b215411d85c3e (DEFAULT_TABLE table=test-table [id=3e4a6578da084984b45a377be1f70a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:05:40.213133 14371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2fd8d3dd634e299d4b215411d85c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:40.213231 14576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2fd8d3dd634e299d4b215411d85c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:40.213301 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2fd8d3dd634e299d4b215411d85c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:40.214926 14922 tablet_bootstrap.cc:492] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d: Bootstrap starting.
I20260805 23:05:40.215504 14916 tablet_bootstrap.cc:492] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: Bootstrap starting.
I20260805 23:05:40.215814 14922 tablet_bootstrap.cc:654] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:40.215860 14917 tablet_bootstrap.cc:492] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: Bootstrap starting.
I20260805 23:05:40.216701 14916 tablet_bootstrap.cc:654] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:40.216758 14917 tablet_bootstrap.cc:654] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:40.217916 14917 tablet_bootstrap.cc:492] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: No bootstrap required, opened a new log
I20260805 23:05:40.217991 14917 ts_tablet_manager.cc:1397] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:40.218343 14917 raft_consensus.cc:348] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.218441 14917 raft_consensus.cc:374] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:40.218470 14917 raft_consensus.cc:729] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acc628c5e8c469ca391c52857e972fe, State: Initialized, Role: FOLLOWER
I20260805 23:05:40.218587 14917 consensus_queue.cc:260] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.218814 14917 ts_tablet_manager.cc:1428] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260805 23:05:40.219273 14916 tablet_bootstrap.cc:492] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: No bootstrap required, opened a new log
I20260805 23:05:40.219338 14916 ts_tablet_manager.cc:1397] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260805 23:05:40.219726 14916 raft_consensus.cc:348] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.219820 14916 raft_consensus.cc:374] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:40.219848 14916 raft_consensus.cc:729] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6ef6eed92842c3a4cf0e11a98660ba, State: Initialized, Role: FOLLOWER
I20260805 23:05:40.219952 14916 consensus_queue.cc:260] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.220175 14916 ts_tablet_manager.cc:1428] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:40.225040 14922 tablet_bootstrap.cc:492] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d: No bootstrap required, opened a new log
I20260805 23:05:40.225138 14922 ts_tablet_manager.cc:1397] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260805 23:05:40.225525 14922 raft_consensus.cc:348] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.225642 14922 raft_consensus.cc:374] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:40.225672 14922 raft_consensus.cc:729] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086cb35b832146f9b01b50dbd9cf4b0d, State: Initialized, Role: FOLLOWER
I20260805 23:05:40.225786 14922 consensus_queue.cc:260] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.226064 14922 ts_tablet_manager.cc:1428] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:05:40.229022 14923 raft_consensus.cc:482] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:05:40.229162 14923 raft_consensus.cc:504] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.230340 14923 leader_election.cc:288] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977), 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:40.235335 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.230545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:41893 (local address 127.3.12.66:48631)
0805 23:05:40.231339 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:40.231354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:40.231432 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:40.231810 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:40.231813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:40.231829 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:40.231956 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:40.231962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.232949 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.232952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:40.234136 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:40.234147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.234842 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.234846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:40.234949 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.234968 (+ 19us) client_negotiation.cc:769] Sending connection context
0805 23:05:40.235074 (+ 106us) client_negotiation.cc:240] Negotiation successful
0805 23:05:40.235145 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":46,"thread_start_us":63,"threads_started":1}
I20260805 23:05:40.235512 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.230545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42977 (local address 127.3.12.66:36907)
0805 23:05:40.231017 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:40.231034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:40.231154 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:40.231422 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:40.231425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:40.231447 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:05:40.231579 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:40.231584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.232706 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.232709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:40.234086 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:40.234098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.235069 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.235073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:40.235205 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.235227 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:05:40.235294 (+ 67us) client_negotiation.cc:240] Negotiation successful
0805 23:05:40.235365 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260805 23:05:40.235554 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.230842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:36907 (local address 127.3.12.65:42977)
0805 23:05:40.230960 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:40.230964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:40.231107 (+ 143us) server_negotiation.cc:407] Connection header received
0805 23:05:40.231258 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:40.231261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:40.231309 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:40.231390 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:40.231703 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.232557 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.234234 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.234844 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.234939 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.234995 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:40.235319 (+ 324us) server_negotiation.cc:299] Negotiation successful
0805 23:05:40.235412 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260805 23:05:40.235782 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.230666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:48631 (local address 127.3.12.67:41893)
0805 23:05:40.230891 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:40.230896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:40.231526 (+ 630us) server_negotiation.cc:407] Connection header received
0805 23:05:40.231578 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:40.231580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:40.231615 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:40.231709 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:05:40.232090 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.232834 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.234264 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.234691 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.234820 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.234888 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:40.235541 (+ 653us) server_negotiation.cc:299] Negotiation successful
0805 23:05:40.235629 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
I20260805 23:05:40.236177 14807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd8d3dd634e299d4b215411d85c3e" candidate_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acc628c5e8c469ca391c52857e972fe" is_pre_election: true
I20260805 23:05:40.236304 14401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd8d3dd634e299d4b215411d85c3e" candidate_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" is_pre_election: true
I20260805 23:05:40.236343 14807 raft_consensus.cc:2454] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6ef6eed92842c3a4cf0e11a98660ba in term 0.
I20260805 23:05:40.236423 14401 raft_consensus.cc:2454] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6ef6eed92842c3a4cf0e11a98660ba in term 0.
I20260805 23:05:40.236786 14511 leader_election.cc:302] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 086cb35b832146f9b01b50dbd9cf4b0d, 7acc628c5e8c469ca391c52857e972fe, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:40.237080 14923 raft_consensus.cc:2781] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:05:40.237140 14923 raft_consensus.cc:482] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:05:40.237200 14923 raft_consensus.cc:3037] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:40.238299 14923 raft_consensus.cc:504] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.238726 14923 leader_election.cc:288] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [CANDIDATE]: Term 1 election: Requested vote from peers 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977), 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893)
I20260805 23:05:40.239082 14807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd8d3dd634e299d4b215411d85c3e" candidate_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acc628c5e8c469ca391c52857e972fe"
I20260805 23:05:40.239081 14401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2fd8d3dd634e299d4b215411d85c3e" candidate_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086cb35b832146f9b01b50dbd9cf4b0d"
I20260805 23:05:40.239202 14807 raft_consensus.cc:3037] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:40.239202 14401 raft_consensus.cc:3037] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:40.240353 14807 raft_consensus.cc:2454] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6ef6eed92842c3a4cf0e11a98660ba in term 1.
I20260805 23:05:40.240353 14401 raft_consensus.cc:2454] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6ef6eed92842c3a4cf0e11a98660ba in term 1.
I20260805 23:05:40.240669 14510 leader_election.cc:302] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 086cb35b832146f9b01b50dbd9cf4b0d, 7b6ef6eed92842c3a4cf0e11a98660ba; no voters:
I20260805 23:05:40.240861 14923 raft_consensus.cc:2781] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:40.241151 14923 raft_consensus.cc:686] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 LEADER]: Becoming Leader. State: Replica: 7b6ef6eed92842c3a4cf0e11a98660ba, State: Running, Role: LEADER
I20260805 23:05:40.241416 14923 consensus_queue.cc:237] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } }
I20260805 23:05:40.243752 14168 catalog_manager.cc:5495] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66). New cstate: current_term: 1 leader_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b6ef6eed92842c3a4cf0e11a98660ba" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 } health_report { overall_health: UNKNOWN } } }
I20260805 23:05:40.267508 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.264035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.3.12.66:41305)
0805 23:05:40.264367 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:40.264374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:40.264386 (+ 12us) server_negotiation.cc:407] Connection header received
0805 23:05:40.264458 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:40.264460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:40.264509 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:40.264615 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:40.264816 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.265358 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.266119 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.266350 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.266425 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.266642 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:40.267166 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:40.267257 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:40.267342 (+ 85us) server_negotiation.cc:299] Negotiation successful
0805 23:05:40.267386 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":129,"threads_started":1}
I20260805 23:05:40.278107 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.274458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42977 (local address 127.3.12.66:59457)
0805 23:05:40.274796 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:40.274814 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:40.274917 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:40.275201 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:40.275204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:40.275211 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:40.275343 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:40.275348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.276079 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.276084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:40.276794 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:40.276804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.277330 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.277334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:40.277432 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.277475 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:40.277794 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:40.277803 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 23:05:40.277863 (+ 60us) client_negotiation.cc:240] Negotiation successful
0805 23:05:40.277944 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260805 23:05:40.278087 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.274734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:59457 (local address 127.3.12.65:42977)
0805 23:05:40.274886 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:40.274890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:40.274905 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:40.275059 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:40.275062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:40.275106 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:40.275189 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:40.275459 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.275960 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.276920 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.277181 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.277307 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.277631 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:40.277704 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:40.277787 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:40.277883 (+ 96us) server_negotiation.cc:299] Negotiation successful
0805 23:05:40.277931 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:05:40.284193 14807 raft_consensus.cc:1261] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Refusing update from remote peer 7b6ef6eed92842c3a4cf0e11a98660ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:40.284188 14401 raft_consensus.cc:1261] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Refusing update from remote peer 7b6ef6eed92842c3a4cf0e11a98660ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:05:40.284767 15043 consensus_queue.cc:1035] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "086cb35b832146f9b01b50dbd9cf4b0d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:40.284988 14923 consensus_queue.cc:1035] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acc628c5e8c469ca391c52857e972fe" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:05:40.302029 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.298422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:41305 (local address 127.3.12.65:58235)
0805 23:05:40.298601 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:40.298619 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:40.298704 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:40.299015 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:40.299018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:40.299027 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0805 23:05:40.299143 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:40.299152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.299924 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.299927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:40.300713 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:40.300723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.301300 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.301304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:40.301393 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.301439 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0805 23:05:40.301724 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0805 23:05:40.301733 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 23:05:40.301799 (+ 66us) client_negotiation.cc:240] Negotiation successful
0805 23:05:40.301883 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260805 23:05:40.301994 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.298666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:58235 (local address 127.3.12.66:41305)
0805 23:05:40.298787 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:40.298791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:40.298803 (+ 12us) server_negotiation.cc:407] Connection header received
0805 23:05:40.298848 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:40.298851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:40.298904 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:40.298980 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:05:40.299281 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.299815 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.300856 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:40.301181 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:40.301264 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:40.301561 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:05:40.301628 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:05:40.301715 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:40.301809 (+ 94us) server_negotiation.cc:299] Negotiation successful
0805 23:05:40.301858 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260805 23:05:40.323452 14170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:40.323606 14170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:05:40.325730 14170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3a813a82c5947dbbf1be0c0a4348fe4: Sending DeleteTablet for 3 replicas of tablet ee2fd8d3dd634e299d4b215411d85c3e
I20260805 23:05:40.326370 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet ee2fd8d3dd634e299d4b215411d85c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:40 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:52098
I20260805 23:05:40.326392 14371 tablet_service.cc:1514] Processing DeleteTablet for tablet ee2fd8d3dd634e299d4b215411d85c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:40 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:45118
I20260805 23:05:40.326578 14947 tablet_replica.cc:330] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: stopping tablet replica
I20260805 23:05:40.326701 14947 raft_consensus.cc:2229] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:40.326735 14576 tablet_service.cc:1514] Processing DeleteTablet for tablet ee2fd8d3dd634e299d4b215411d85c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:05:40 UTC) from {username='oryx', principal='oryx/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:48146
I20260805 23:05:40.326820 14947 raft_consensus.cc:2258] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:40.326993 14949 tablet_replica.cc:330] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d: stopping tablet replica
I20260805 23:05:40.327119 14949 raft_consensus.cc:2229] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:05:40.327239 14949 raft_consensus.cc:2258] T ee2fd8d3dd634e299d4b215411d85c3e P 086cb35b832146f9b01b50dbd9cf4b0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:40.327369 14952 tablet_replica.cc:330] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: stopping tablet replica
I20260805 23:05:40.327486 14952 raft_consensus.cc:2229] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:05:40.327519 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 14285
I20260805 23:05:40.327823 14952 raft_consensus.cc:2258] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:05:40.328722 14947 ts_tablet_manager.cc:1905] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:40.331391 14947 ts_tablet_manager.cc:1918] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:40.331481 14947 log.cc:1196] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/ee2fd8d3dd634e299d4b215411d85c3e
I20260805 23:05:40.331753 14947 ts_tablet_manager.cc:1939] T ee2fd8d3dd634e299d4b215411d85c3e P 7acc628c5e8c469ca391c52857e972fe: Deleting consensus metadata
I20260805 23:05:40.332527 14149 catalog_manager.cc:4841] TS 7acc628c5e8c469ca391c52857e972fe (127.3.12.67:41893): tablet ee2fd8d3dd634e299d4b215411d85c3e (table test-table [id=3e4a6578da084984b45a377be1f70a5a]) successfully deleted
I20260805 23:05:40.337122 14952 ts_tablet_manager.cc:1905] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:05:40.340281 14952 ts_tablet_manager.cc:1918] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260805 23:05:40.340399 14952 log.cc:1196] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/ee2fd8d3dd634e299d4b215411d85c3e
I20260805 23:05:40.340773 14952 ts_tablet_manager.cc:1939] T ee2fd8d3dd634e299d4b215411d85c3e P 7b6ef6eed92842c3a4cf0e11a98660ba: Deleting consensus metadata
I20260805 23:05:40.341572 14148 catalog_manager.cc:4841] TS 7b6ef6eed92842c3a4cf0e11a98660ba (127.3.12.66:41305): tablet ee2fd8d3dd634e299d4b215411d85c3e (table test-table [id=3e4a6578da084984b45a377be1f70a5a]) successfully deleted
W20260805 23:05:40.350638 14148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:42977 (error 108)
I20260805 23:05:40.351016 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 14482
I20260805 23:05:40.360816 15057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:40.358021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42977 (local address 127.0.0.1:45142)
0805 23:05:40.360534 (+ 2513us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:40.360654 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2173,"thread_start_us":85,"threads_started":1}
W20260805 23:05:40.360996 14148 catalog_manager.cc:4584] TS 086cb35b832146f9b01b50dbd9cf4b0d (127.3.12.65:42977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee2fd8d3dd634e299d4b215411d85c3e: Network error: Client connection negotiation failed: client connection to 127.3.12.65:42977: connect: Connection refused (error 111)
I20260805 23:05:40.368095 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 14689
I20260805 23:05:40.384951 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 14128
2026-08-05T23:05:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3497 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:40 dist-test-slave-2d19 krb5kdc[15061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:40 dist-test-slave-2d19 krb5kdc[15061](info): set up 2 sockets
Aug 05 23:05:40 dist-test-slave-2d19 krb5kdc[15061](info): commencing operation
krb5kdc: starting...
W20260805 23:05:41.609728 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:41 dist-test-slave-2d19 krb5kdc[15061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:41.743036 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42455
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:42455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:41.857297 15080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:41.857653 15080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:41.857730 15080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:41.861955 15080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:41.862066 15080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:41.862103 15080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:41.862146 15080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:41.862185 15080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:41.868019 15080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:42455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:41.869494 15080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:41.870702 15080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:05:41.880080 15080 server_base.cc:1034] running on GCE node
W20260805 23:05:41.880024 15089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:41.880036 15087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:41.880024 15091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:41.880945 15080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:41.882131 15080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:41.883347 15080 hybrid_clock.cc:648] HybridClock initialized: now 1785971141883332 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-05T23:05:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1500 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:05:41 dist-test-slave-2d19 krb5kdc[15100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:05:41 dist-test-slave-2d19 krb5kdc[15100](info): set up 2 sockets
Aug 05 23:05:41 dist-test-slave-2d19 krb5kdc[15100](info): commencing operation
krb5kdc: starting...
W20260805 23:05:43.133921 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:05:43 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:05:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:05:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:05:43.275614 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43213
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:43213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:43.391139 15123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:43.391439 15123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:43.391484 15123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:43.395618 15123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:43.395694 15123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:43.395713 15123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:43.395731 15123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:43.395746 15123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:43.401283 15123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:43213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:43.402420 15123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:43.403538 15123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:43.412526 15134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:43.412591 15132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:43.412608 15131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:43.412824 15123 server_base.cc:1034] running on GCE node
I20260805 23:05:43.413522 15123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:43.414637 15123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:43.415840 15123 hybrid_clock.cc:648] HybridClock initialized: now 1785971143415821 us; error 32 us; skew 500 ppm
Aug 05 23:05:43 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:43.424486 15123 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:43.425500 15123 webserver.cc:460] Webserver started at http://127.3.12.126:41659/ using document root <none> and password file <none>
I20260805 23:05:43.426160 15123 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:43.426263 15123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:43.426515 15123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:43.428745 15123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "4e34795b870b4811a9a1308ac6386554"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "3e4d6e3f8b885fd37514eabf21abfa3b"
server_key_iv: "1febb90e293fa7f088d7653bbd38a152"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.429359 15123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "4e34795b870b4811a9a1308ac6386554"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "3e4d6e3f8b885fd37514eabf21abfa3b"
server_key_iv: "1febb90e293fa7f088d7653bbd38a152"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.433423 15123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:43.436053 15143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:43.437525 15123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:43.437696 15123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "4e34795b870b4811a9a1308ac6386554"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "3e4d6e3f8b885fd37514eabf21abfa3b"
server_key_iv: "1febb90e293fa7f088d7653bbd38a152"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.437821 15123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:43.467445 15123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:43.471122 15123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:43.471341 15123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:43.478638 15123 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:43213
I20260805 23:05:43.478756 15217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:43213 every 8 connection(s)
I20260805 23:05:43.479704 15123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:43.482746 15218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:05:43.483781 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 15123
I20260805 23:05:43.483916 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:05:43.484279 3121 external_mini_cluster.cc:1428] Setting key 14674415a1a275f95f3ec0950b81d011
I20260805 23:05:43.490162 15218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Bootstrap starting.
I20260805 23:05:43.492944 15218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Neither blocks nor log segments found. Creating new log.
I20260805 23:05:43.493721 15218 log.cc:824] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:05:43 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:43.496292 15218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: No bootstrap required, opened a new log
I20260805 23:05:43.500051 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:43.485741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.3.12.126:43213)
0805 23:05:43.486183 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:43.486206 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:43.486252 (+ 46us) server_negotiation.cc:407] Connection header received
0805 23:05:43.487096 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:43.487120 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:43.487564 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:43.487936 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:43.488763 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:43.490223 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:43.491141 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:43.491859 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:43.492040 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:43.494708 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:43.494715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:43.494731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:43.494762 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:43.496780 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:43.497239 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:43.497245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:43.497252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:43.497395 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:43.497667 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:43.497671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:43.497673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:43.498242 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:43.498487 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:43.498738 (+ 251us) server_negotiation.cc:299] Negotiation successful
0805 23:05:43.499073 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":106,"threads_started":1}
I20260805 23:05:43.500990 15218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } }
I20260805 23:05:43.501276 15218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:05:43.501343 15218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e34795b870b4811a9a1308ac6386554, State: Initialized, Role: FOLLOWER
I20260805 23:05:43.501823 15218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } }
I20260805 23:05:43.501994 15218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:43.502084 15218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:43.502187 15218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:05:43.503225 15218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } }
I20260805 23:05:43.503585 15218 leader_election.cc:302] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e34795b870b4811a9a1308ac6386554; no voters:
I20260805 23:05:43.503952 15218 leader_election.cc:288] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:05:43.504034 15224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:05:43.504390 15224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 1 LEADER]: Becoming Leader. State: Replica: 4e34795b870b4811a9a1308ac6386554, State: Running, Role: LEADER
I20260805 23:05:43.504703 15224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } }
I20260805 23:05:43.505399 15218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:43.506601 15225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e34795b870b4811a9a1308ac6386554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } } }
I20260805 23:05:43.506611 15232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e34795b870b4811a9a1308ac6386554. Latest consensus state: current_term: 1 leader_uuid: "4e34795b870b4811a9a1308ac6386554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } } }
I20260805 23:05:43.506990 15225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:43.507022 15232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: This master's current role is: LEADER
W20260805 23:05:43.510679 15254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:05:43.510748 15254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:05:43.510833 15247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:43.511514 15247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:43.517117 15247 catalog_manager.cc:1347] Generated new cluster ID: e275641c0eb54f32b167f8062dcd8e13
I20260805 23:05:43.517206 15247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:43.535482 15247 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:05:43.536903 15247 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:43.555948 15247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Generated new TSK 0
I20260805 23:05:43.556958 15247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:05:43.587982 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43213
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:43.704704 15261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:43.705049 15261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:43.705128 15261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:43.709487 15261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:43.709549 15261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:43.709609 15261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:43.715122 15261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:43.716217 15261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:43.717283 15261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:43.727033 15266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:43.726976 15269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:43.726976 15267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:43.727361 15261 server_base.cc:1034] running on GCE node
I20260805 23:05:43.727810 15261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:43.728493 15261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:43.729655 15261 hybrid_clock.cc:648] HybridClock initialized: now 1785971143729641 us; error 26 us; skew 500 ppm
Aug 05 23:05:43 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:43.732616 15261 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:05:43.733565 15261 webserver.cc:460] Webserver started at http://127.3.12.65:36401/ using document root <none> and password file <none>
I20260805 23:05:43.734135 15261 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:43.734230 15261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:43.734473 15261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:43.736581 15261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "cba5c211baea4654956a4df4e285fe08"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "8e9828392b4218801fec9d88c4006539"
server_key_iv: "40e09ef460f946488e0d185561d2fdcc"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.737143 15261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "cba5c211baea4654956a4df4e285fe08"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "8e9828392b4218801fec9d88c4006539"
server_key_iv: "40e09ef460f946488e0d185561d2fdcc"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.740922 15261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260805 23:05:43.743417 15281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:43.744659 15261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260805 23:05:43.744807 15261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "cba5c211baea4654956a4df4e285fe08"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "8e9828392b4218801fec9d88c4006539"
server_key_iv: "40e09ef460f946488e0d185561d2fdcc"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.744921 15261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:43.764113 15261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:43.767215 15261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:43.767424 15261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:43.768033 15261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:43.768977 15261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:43.769054 15261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:43.769132 15261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:43.769178 15261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:43.778546 15261 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:37419
I20260805 23:05:43.778566 15442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:37419 every 8 connection(s)
I20260805 23:05:43.779495 15261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:43.786338 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 15261
I20260805 23:05:43.786527 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:05:43.786845 3121 external_mini_cluster.cc:1428] Setting key a4b20213016832aa35c6b7a2ee2a4f13
Aug 05 23:05:43 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:43.794637 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:43.781444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:38859 (local address 127.3.12.126:43213)
0805 23:05:43.781645 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:43.781649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:43.782670 (+ 1021us) server_negotiation.cc:407] Connection header received
0805 23:05:43.784070 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:43.784074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:43.784133 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:43.784244 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:43.785475 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:43.786042 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:43.787439 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:43.787764 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:43.787862 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:43.790903 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:43.790907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:43.790910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:43.790944 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:43.792538 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:43.793068 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:43.793072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:43.793074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:43.793158 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:43.793519 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:43.793523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:43.793525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:43.793714 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:43.793826 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:43.794335 (+ 509us) server_negotiation.cc:299] Negotiation successful
0805 23:05:43.794461 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260805 23:05:43.795686 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:43.781866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:38859)
0805 23:05:43.782384 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:43.782430 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:43.783794 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:43.784454 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:43.784462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:43.784834 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:43.785316 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:43.785329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:43.786183 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:43.786192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:43.787271 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:43.787284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:43.787935 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:43.787939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:43.788157 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:43.788923 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:43.788941 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:43.790718 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:43.792664 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:43.792670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:43.792687 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:43.792957 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:43.793257 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:43.793261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:43.793263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:43.793411 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:43.793827 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:43.793835 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:43.794095 (+ 260us) client_negotiation.cc:769] Sending connection context
0805 23:05:43.794292 (+ 197us) client_negotiation.cc:240] Negotiation successful
0805 23:05:43.794644 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":119,"threads_started":1}
I20260805 23:05:43.797209 15443 heartbeater.cc:347] Connected to a master server at 127.3.12.126:43213
I20260805 23:05:43.797497 15443 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:05:43.798133 15443 heartbeater.cc:511] Master 127.3.12.126:43213 requested a full tablet report, sending...
I20260805 23:05:43.800086 15168 ts_manager.cc:194] Registered new tserver with Master: cba5c211baea4654956a4df4e285fe08 (127.3.12.65:37419)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:05:43.801414 15168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:38859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:05:43.820631 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43213
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:43.938175 15450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:43.938444 15450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:43.938496 15450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:43.942730 15450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:43.942802 15450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:43.942865 15450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:43.948653 15450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:43.949857 15450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:43.950939 15450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:43.961133 15456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:43.961304 15450 server_base.cc:1034] running on GCE node
W20260805 23:05:43.961133 15457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:43.961516 15461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:43.961903 15450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:43.962598 15450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:43.963799 15450 hybrid_clock.cc:648] HybridClock initialized: now 1785971143963778 us; error 34 us; skew 500 ppm
Aug 05 23:05:43 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:43.966679 15450 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:05:43.967640 15450 webserver.cc:460] Webserver started at http://127.3.12.66:43653/ using document root <none> and password file <none>
I20260805 23:05:43.968222 15450 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:43.968376 15450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:43.968678 15450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:43.970767 15450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "c62c78654dbd4a3db3300252c93d246b"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "d2148417dfe3dd4015a92768d9a44ab6"
server_key_iv: "8093f63ff6b440214a9825b5fcd92a16"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.971350 15450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "c62c78654dbd4a3db3300252c93d246b"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "d2148417dfe3dd4015a92768d9a44ab6"
server_key_iv: "8093f63ff6b440214a9825b5fcd92a16"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.975158 15450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260805 23:05:43.977679 15470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:43.978899 15450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:05:43.979081 15450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "c62c78654dbd4a3db3300252c93d246b"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "d2148417dfe3dd4015a92768d9a44ab6"
server_key_iv: "8093f63ff6b440214a9825b5fcd92a16"
server_key_version: "encryptionkey@0"
I20260805 23:05:43.979200 15450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:43.999720 15450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:44.002969 15450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:44.003242 15450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:44.003892 15450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:44.004851 15450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:44.004927 15450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.005009 15450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:44.005062 15450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.014786 15450 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:44921
I20260805 23:05:44.014804 15640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:44921 every 8 connection(s)
I20260805 23:05:44.015730 15450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:44.018494 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 15450
I20260805 23:05:44.018639 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:05:44.018998 3121 external_mini_cluster.cc:1428] Setting key f83eae3df5c9f76a3f830d42f38e609c
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:05:44.032295 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:44.017705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:53699 (local address 127.3.12.126:43213)
0805 23:05:44.017868 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:44.017872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:44.018854 (+ 982us) server_negotiation.cc:407] Connection header received
0805 23:05:44.020412 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:44.020415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:44.020471 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:44.020560 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:44.021796 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.022656 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.024580 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.024959 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.025089 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.028080 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:44.028087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:44.028094 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:44.028134 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:44.029970 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:44.030537 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:44.030541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:44.030543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:44.030631 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:44.031013 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:44.031017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:44.031019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:44.031277 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:44.031388 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:44.031943 (+ 555us) server_negotiation.cc:299] Negotiation successful
0805 23:05:44.032096 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 23:05:44.033398 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:44.018152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:53699)
0805 23:05:44.018670 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:44.018725 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:44.020151 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:44.020768 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:44.020775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:44.021148 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:44.021642 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.021655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.022819 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.022825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.024402 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.024412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.025124 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.025127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.025276 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.025936 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:44.025955 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:44.027871 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:44.030114 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:44.030120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:44.030135 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:44.030411 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:44.030735 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:44.030738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:44.030740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:44.030893 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:44.031387 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:44.031396 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:44.031675 (+ 279us) client_negotiation.cc:769] Sending connection context
0805 23:05:44.031883 (+ 208us) client_negotiation.cc:240] Negotiation successful
0805 23:05:44.032541 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
I20260805 23:05:44.034870 15642 heartbeater.cc:347] Connected to a master server at 127.3.12.126:43213
I20260805 23:05:44.035218 15642 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:44.035887 15642 heartbeater.cc:511] Master 127.3.12.126:43213 requested a full tablet report, sending...
I20260805 23:05:44.037323 15168 ts_manager.cc:194] Registered new tserver with Master: c62c78654dbd4a3db3300252c93d246b (127.3.12.66:44921)
I20260805 23:05:44.038239 15168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:53699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:05:44.053755 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43213
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:44.169386 15650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:44.169682 15650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:44.169764 15650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:44.173873 15650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:44.173971 15650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:44.174074 15650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:44.179360 15650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:44.180518 15650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:44.181564 15650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:44.191668 15656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:44.191670 15658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:44.191696 15660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:44.191890 15650 server_base.cc:1034] running on GCE node
I20260805 23:05:44.192515 15650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:44.193169 15650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:44.194339 15650 hybrid_clock.cc:648] HybridClock initialized: now 1785971144194325 us; error 26 us; skew 500 ppm
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:44.197285 15650 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:05:44.198150 15650 webserver.cc:460] Webserver started at http://127.3.12.67:41665/ using document root <none> and password file <none>
I20260805 23:05:44.198675 15650 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:44.198767 15650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:44.198939 15650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:05:44.201038 15650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "8ea4407864da4500941cd307fc9bb88e"
format_stamp: "Formatted at 2026-08-05 23:05:44 on dist-test-slave-2d19"
server_key: "6e92e90bac7a82ff256783ed8962f0d9"
server_key_iv: "63a483848f68ca4a7cce44e1d0dd7e3f"
server_key_version: "encryptionkey@0"
I20260805 23:05:44.201545 15650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "8ea4407864da4500941cd307fc9bb88e"
format_stamp: "Formatted at 2026-08-05 23:05:44 on dist-test-slave-2d19"
server_key: "6e92e90bac7a82ff256783ed8962f0d9"
server_key_iv: "63a483848f68ca4a7cce44e1d0dd7e3f"
server_key_version: "encryptionkey@0"
I20260805 23:05:44.205304 15650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:05:44.207813 15667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.209182 15650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:05:44.209285 15650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "8ea4407864da4500941cd307fc9bb88e"
format_stamp: "Formatted at 2026-08-05 23:05:44 on dist-test-slave-2d19"
server_key: "6e92e90bac7a82ff256783ed8962f0d9"
server_key_iv: "63a483848f68ca4a7cce44e1d0dd7e3f"
server_key_version: "encryptionkey@0"
I20260805 23:05:44.209370 15650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:44.219527 15650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:44.222479 15650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:44.222630 15650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:44.223170 15650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:44.224020 15650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:44.224082 15650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.224143 15650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:44.224390 15650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260805 23:05:44.233980 15650 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:46639
I20260805 23:05:44.234001 15836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:46639 every 8 connection(s)
I20260805 23:05:44.234918 15650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:44.241307 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 15650
I20260805 23:05:44.241447 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:05:44.241742 3121 external_mini_cluster.cc:1428] Setting key 44b8c3218650a8d50f4da9c7a348daf3
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:05:44.249694 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:44.236976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:48529 (local address 127.3.12.126:43213)
0805 23:05:44.237127 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:44.237131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:44.238076 (+ 945us) server_negotiation.cc:407] Connection header received
0805 23:05:44.239298 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:44.239301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:44.239351 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:44.239433 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:44.240678 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.241468 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.242918 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.243221 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.243352 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.245949 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:44.245952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:44.245954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:44.245984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:44.247368 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:44.248001 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:44.248004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:44.248005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:44.248092 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:44.248512 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:44.248515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:44.248517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:44.248794 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:44.248884 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:44.249446 (+ 562us) server_negotiation.cc:299] Negotiation successful
0805 23:05:44.249556 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:05:44.250604 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:44.237370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:48529)
0805 23:05:44.237833 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:44.237878 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:44.239078 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:44.239605 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:44.239612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:44.239971 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:44.240516 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.240529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.241606 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.241609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.242817 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.242827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.243394 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.243399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.243592 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.244234 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:44.244306 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:44.245779 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:05:44.247577 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:44.247586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:44.247612 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:44.247883 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:44.248186 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:05:44.248188 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:05:44.248189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:05:44.248381 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:05:44.248899 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:05:44.248904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:05:44.249189 (+ 285us) client_negotiation.cc:769] Sending connection context
0805 23:05:44.249347 (+ 158us) client_negotiation.cc:240] Negotiation successful
0805 23:05:44.249649 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260805 23:05:44.251935 15837 heartbeater.cc:347] Connected to a master server at 127.3.12.126:43213
I20260805 23:05:44.252195 15837 heartbeater.cc:464] Registering TS with master...
I20260805 23:05:44.252748 15837 heartbeater.cc:511] Master 127.3.12.126:43213 requested a full tablet report, sending...
I20260805 23:05:44.253847 15168 ts_manager.cc:194] Registered new tserver with Master: 8ea4407864da4500941cd307fc9bb88e (127.3.12.67:46639)
I20260805 23:05:44.254451 15168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:48529
I20260805 23:05:44.256134 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab.
Entry for principal oryx/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab.
WARNING: no policy specified for oryx/127.3.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab.
Entry for principal oryx/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab.
WARNING: no policy specified for oryx/127.3.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab.
Entry for principal oryx/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab.
I20260805 23:05:44.348174 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 15261
I20260805 23:05:44.363924 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 15450
I20260805 23:05:44.381609 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 15650
I20260805 23:05:44.403805 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 15123
I20260805 23:05:44.414120 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43213
--webserver_interface=127.3.12.126
--webserver_port=41659
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:43213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:05:44.529079 15851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:44.529444 15851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:44.529522 15851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:44.533712 15851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:05:44.533811 15851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:44.533847 15851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:44.533890 15851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:05:44.533926 15851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:05:44.539603 15851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:43213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:44.540822 15851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:44.541896 15851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:44.550958 15865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:44.550974 15862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:44.550974 15861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:44.551231 15851 server_base.cc:1034] running on GCE node
I20260805 23:05:44.551894 15851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:44.553215 15851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:44.554441 15851 hybrid_clock.cc:648] HybridClock initialized: now 1785971144554426 us; error 38 us; skew 500 ppm
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:44.557641 15851 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:05:44.558579 15851 webserver.cc:460] Webserver started at http://127.3.12.126:41659/ using document root <none> and password file <none>
I20260805 23:05:44.559149 15851 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:44.559230 15851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:44.563408 15851 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260805 23:05:44.565691 15876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.567055 15851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 23:05:44.567193 15851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "4e34795b870b4811a9a1308ac6386554"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "3e4d6e3f8b885fd37514eabf21abfa3b"
server_key_iv: "1febb90e293fa7f088d7653bbd38a152"
server_key_version: "encryptionkey@0"
I20260805 23:05:44.567735 15851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:44.601255 15851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:44.604748 15851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:44.604983 15851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:44.612969 15851 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:43213
I20260805 23:05:44.613000 15956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:43213 every 8 connection(s)
I20260805 23:05:44.614167 15851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:05:44.618779 15957 sys_catalog.cc:263] Verifying existing consensus state
I20260805 23:05:44.620527 15957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Bootstrap starting.
I20260805 23:05:44.622118 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 15851
I20260805 23:05:44.622639 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:37419
--local_ip_for_outbound_sockets=127.3.12.65
--tserver_master_addrs=127.3.12.126:43213
--webserver_port=36401
--webserver_interface=127.3.12.65
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.12.65
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:05:44.638757 15957 log.cc:824] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Log is configured to *not* fsync() on all Append() calls
I20260805 23:05:44.645717 15957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:05:44.646117 15957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Bootstrap complete.
I20260805 23:05:44.650573 15957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } }
I20260805 23:05:44.650990 15957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e34795b870b4811a9a1308ac6386554, State: Initialized, Role: FOLLOWER
I20260805 23:05:44.651551 15957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } }
I20260805 23:05:44.651688 15957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:05:44.651755 15957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:05:44.651842 15957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 1 FOLLOWER]: Advancing to term 2
I20260805 23:05:44.653924 15957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } }
I20260805 23:05:44.654261 15957 leader_election.cc:302] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e34795b870b4811a9a1308ac6386554; no voters:
I20260805 23:05:44.654609 15957 leader_election.cc:288] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260805 23:05:44.654708 15963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 23:05:44.654947 15963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [term 2 LEADER]: Becoming Leader. State: Replica: 4e34795b870b4811a9a1308ac6386554, State: Running, Role: LEADER
I20260805 23:05:44.655253 15963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } }
I20260805 23:05:44.655953 15957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:05:44.656694 15964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4e34795b870b4811a9a1308ac6386554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } } }
I20260805 23:05:44.656834 15964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:44.657474 15965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e34795b870b4811a9a1308ac6386554. Latest consensus state: current_term: 2 leader_uuid: "4e34795b870b4811a9a1308ac6386554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e34795b870b4811a9a1308ac6386554" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43213 } } }
I20260805 23:05:44.657634 15965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554 [sys.catalog]: This master's current role is: LEADER
I20260805 23:05:44.657883 15974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:05:44.661608 15974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:05:44.662283 15974 catalog_manager.cc:1259] Loaded cluster ID: e275641c0eb54f32b167f8062dcd8e13
I20260805 23:05:44.662351 15974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:05:44.664750 15974 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:05:44.665448 15974 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e34795b870b4811a9a1308ac6386554: Loaded TSK: 0
I20260805 23:05:44.666702 15974 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 23:05:44.745167 15960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:44.745422 15960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:44.745467 15960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:44.749836 15960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:44.749902 15960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:44.749984 15960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:05:44.755407 15960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.65.keytab
--principal=oryx/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:37419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=36401
--tserver_master_addrs=127.3.12.126:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:44.756524 15960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:44.757614 15960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:44.767597 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:44.767632 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:44.767904 15960 server_base.cc:1034] running on GCE node
W20260805 23:05:44.768081 15993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:44.768536 15960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:44.769263 15960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:44.770437 15960 hybrid_clock.cc:648] HybridClock initialized: now 1785971144770416 us; error 35 us; skew 500 ppm
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971144, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:44.773691 15960 init.cc:377] Logged in from keytab as oryx/127.3.12.65@KRBTEST.COM (short username oryx)
I20260805 23:05:44.774610 15960 webserver.cc:460] Webserver started at http://127.3.12.65:36401/ using document root <none> and password file <none>
I20260805 23:05:44.775183 15960 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:44.775264 15960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:44.779431 15960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:44.781864 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.783221 15960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 23:05:44.783365 15960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "cba5c211baea4654956a4df4e285fe08"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "8e9828392b4218801fec9d88c4006539"
server_key_iv: "40e09ef460f946488e0d185561d2fdcc"
server_key_version: "encryptionkey@0"
I20260805 23:05:44.783944 15960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:44.797245 15960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:44.800455 15960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:44.800664 15960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:44.801262 15960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:44.802103 15960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:44.802187 15960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.802280 15960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:44.802331 15960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.811630 15960 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:37419
I20260805 23:05:44.811661 16177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:37419 every 8 connection(s)
I20260805 23:05:44.812738 15960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:05:44.820781 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 15960
I20260805 23:05:44.821241 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:44921
--local_ip_for_outbound_sockets=127.3.12.66
--tserver_master_addrs=127.3.12.126:43213
--webserver_port=43653
--webserver_interface=127.3.12.66
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.12.66
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260805 23:05:44.826058 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:44.814836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:56541 (local address 127.3.12.126:43213)
0805 23:05:44.815310 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:44.815330 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:44.815625 (+ 295us) server_negotiation.cc:407] Connection header received
0805 23:05:44.817169 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:44.817180 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:44.817463 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:44.817740 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:44.818816 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.819781 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.821043 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.821587 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.821775 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.825298 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:56541: BlockingRecv error: recv got EOF from 127.3.12.65:56541 (error 108)
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":116,"threads_started":1}
I20260805 23:05:44.826072 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:44.814984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:56541)
0805 23:05:44.815454 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:44.815504 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:44.816782 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:44.817839 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:44.817845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:44.818193 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:44.818658 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.818672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.819917 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.819920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.820859 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.820874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.821770 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.821775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.821969 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.822608 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:44.822625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:44.825079 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":133,"threads_started":1}
W20260805 23:05:44.826789 16178 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:43213 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.12.126:43213: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:44.832607 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:44.827358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:54779)
0805 23:05:44.827476 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:44.827492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:44.827576 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:44.827848 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:44.827851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:44.828055 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:44.828149 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.828154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.828914 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.828919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.829454 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.829463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.829910 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.829913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.830016 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.830536 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:44.830548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:44.832458 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260805 23:05:44.832705 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:44.827430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:54779 (local address 127.3.12.126:43213)
0805 23:05:44.827553 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:44.827557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:44.827570 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:05:44.827690 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:44.827693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:44.827747 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:44.827834 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:44.828285 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.828799 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.829579 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.829798 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.829909 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.832592 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:54779: BlockingRecv error: recv got EOF from 127.3.12.65:54779 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:44.839072 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:44.833408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:52163)
0805 23:05:44.833565 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:44.833578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:44.833647 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:44.833871 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:44.833874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:44.834049 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:44.834144 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.834150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.834886 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.834890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.835794 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:44.835804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.836442 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.836446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:44.836539 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.837021 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:44.837037 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:44.838926 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260805 23:05:44.839237 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:44.833525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:52163 (local address 127.3.12.126:43213)
0805 23:05:44.833640 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:44.833643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:44.833655 (+ 12us) server_negotiation.cc:407] Connection header received
0805 23:05:44.833727 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:44.833729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:44.833763 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:44.833827 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:44.834282 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.834782 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.836037 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:44.836313 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:44.836407 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:44.839085 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:52163: BlockingRecv error: recv got EOF from 127.3.12.65:52163 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260805 23:05:44.839490 16178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 23:05:44.940156 16186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:44.940533 16186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:44.940580 16186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:44.944664 16186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:44.944730 16186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:44.944790 16186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:05:44.950320 16186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.66.keytab
--principal=oryx/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:44921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=43653
--tserver_master_addrs=127.3.12.126:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:44.951493 16186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:44.952617 16186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:44.963147 16196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:44.963150 16195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:44.963189 16186 server_base.cc:1034] running on GCE node
W20260805 23:05:44.963150 16200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:44.963868 16186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:44.964627 16186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:44.965801 16186 hybrid_clock.cc:648] HybridClock initialized: now 1785971144965785 us; error 25 us; skew 500 ppm
Aug 05 23:05:44 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971144, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:44.969026 16186 init.cc:377] Logged in from keytab as oryx/127.3.12.66@KRBTEST.COM (short username oryx)
I20260805 23:05:44.970036 16186 webserver.cc:460] Webserver started at http://127.3.12.66:43653/ using document root <none> and password file <none>
I20260805 23:05:44.970638 16186 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:44.970714 16186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:44.974959 16186 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:44.977240 16212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:44.978567 16186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:05:44.978716 16186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "c62c78654dbd4a3db3300252c93d246b"
format_stamp: "Formatted at 2026-08-05 23:05:43 on dist-test-slave-2d19"
server_key: "d2148417dfe3dd4015a92768d9a44ab6"
server_key_iv: "8093f63ff6b440214a9825b5fcd92a16"
server_key_version: "encryptionkey@0"
I20260805 23:05:44.979326 16186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:44.998186 16186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:45.001737 16186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:45.001986 16186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:45.002645 16186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:45.003543 16186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:45.003621 16186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:45.003697 16186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:45.003743 16186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:45.013515 16186 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:44921
I20260805 23:05:45.013576 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:44921 every 8 connection(s)
I20260805 23:05:45.014561 16186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:05:45.019547 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 16186
I20260805 23:05:45.020025 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:46639
--local_ip_for_outbound_sockets=127.3.12.67
--tserver_master_addrs=127.3.12.126:43213
--webserver_port=41665
--webserver_interface=127.3.12.67
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.12.67
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260805 23:05:45.026714 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:45.016621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:35397 (local address 127.3.12.126:43213)
0805 23:05:45.016799 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:45.016803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:45.017649 (+ 846us) server_negotiation.cc:407] Connection header received
0805 23:05:45.019080 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:45.019085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:45.019155 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:45.019290 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:45.020662 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.021201 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.022358 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.022582 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.022695 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.026561 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:35397: BlockingRecv error: recv got EOF from 127.3.12.66:35397 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260805 23:05:45.027360 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:45.016993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:35397)
0805 23:05:45.017472 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:45.017524 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:45.018810 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:45.019468 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:45.019475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:45.019906 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:45.020504 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.020521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.021334 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.021340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.022225 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.022235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.022740 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.022745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.022921 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.023585 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:45.023603 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:45.026426 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
W20260805 23:05:45.027885 16372 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:43213 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.12.126:43213: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:45.033672 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:45.028401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:43529)
0805 23:05:45.028524 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:45.028541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:45.028655 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:45.028932 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:45.028935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:45.029103 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:45.029197 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.029202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.029939 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.029943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.030494 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.030503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.030987 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.030991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.031083 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.031613 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:45.031625 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:45.033574 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260805 23:05:45.033951 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:45.028476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:43529 (local address 127.3.12.126:43213)
0805 23:05:45.028602 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:45.028607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:45.028622 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:45.028773 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:45.028776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:45.028827 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:45.028914 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:45.029314 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.029826 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.030652 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.030870 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.030984 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.033830 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:43529: BlockingRecv error: recv got EOF from 127.3.12.66:43529 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:45.040133 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:45.034570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:46745)
0805 23:05:45.034774 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:45.034787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:45.034864 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:45.035138 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:45.035140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:45.035266 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:45.035341 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.035346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.036370 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.036373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.036906 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.036913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.037482 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.037485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.037544 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.038076 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:45.038088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:45.040030 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140}
W20260805 23:05:45.040411 16372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 23:05:45.040458 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:45.034663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:46745 (local address 127.3.12.126:43213)
0805 23:05:45.034776 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:45.034780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:45.034849 (+ 69us) server_negotiation.cc:407] Connection header received
0805 23:05:45.034942 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:45.034944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:45.035006 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:45.035088 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:45.035444 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.036245 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.037033 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.037373 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.037467 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.040194 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:46745: BlockingRecv error: recv got EOF from 127.3.12.66:46745 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260805 23:05:45.138705 16380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:05:45.138990 16380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:05:45.139036 16380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:05:45.143380 16380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:05:45.143447 16380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:05:45.143512 16380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:05:45.149473 16380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/krb5kdc/oryx_127.3.12.67.keytab
--principal=oryx/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:46639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=41665
--tserver_master_addrs=127.3.12.126:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:05:45.150542 16380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:05:45.151592 16380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:05:45.161630 16389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:05:45.161643 16394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:45.161756 16380 server_base.cc:1034] running on GCE node
W20260805 23:05:45.161646 16387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:05:45.162334 16380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:05:45.162950 16380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:05:45.164152 16380 hybrid_clock.cc:648] HybridClock initialized: now 1785971145164137 us; error 30 us; skew 500 ppm
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971145, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:05:45.166898 16380 init.cc:377] Logged in from keytab as oryx/127.3.12.67@KRBTEST.COM (short username oryx)
I20260805 23:05:45.167927 16380 webserver.cc:460] Webserver started at http://127.3.12.67:41665/ using document root <none> and password file <none>
I20260805 23:05:45.168561 16380 fs_manager.cc:359] Metadata directory not provided
I20260805 23:05:45.168676 16380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:05:45.172854 16380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:05:45.174903 16405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:45.176194 16380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:05:45.176357 16380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "8ea4407864da4500941cd307fc9bb88e"
format_stamp: "Formatted at 2026-08-05 23:05:44 on dist-test-slave-2d19"
server_key: "6e92e90bac7a82ff256783ed8962f0d9"
server_key_iv: "63a483848f68ca4a7cce44e1d0dd7e3f"
server_key_version: "encryptionkey@0"
I20260805 23:05:45.176905 16380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:05:45.195935 16380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:05:45.199396 16380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:05:45.199627 16380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:05:45.200349 16380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:05:45.201249 16380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:05:45.201324 16380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:45.201400 16380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:05:45.201444 16380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:05:45.210824 16380 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:46639
I20260805 23:05:45.210858 16569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:46639 every 8 connection(s)
I20260805 23:05:45.211851 16380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:05:45.218248 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 16380
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260805 23:05:45.223886 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:45.213852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:51687 (local address 127.3.12.126:43213)
0805 23:05:45.214030 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:45.214034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:45.214868 (+ 834us) server_negotiation.cc:407] Connection header received
0805 23:05:45.216303 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:45.216306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:45.216362 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:45.216477 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:45.217641 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.218166 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.219505 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.219840 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.219920 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.223716 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:51687: BlockingRecv error: recv got EOF from 127.3.12.67:51687 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260805 23:05:45.224464 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:45.214265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:51687)
0805 23:05:45.214715 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:45.214757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:45.216060 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:45.216658 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:45.216665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:45.217027 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:45.217488 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.217502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.218297 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.218300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.219332 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.219345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.220011 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.220018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.220278 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.220964 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:45.220980 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:45.223544 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":124,"threads_started":1}
W20260805 23:05:45.225102 16572 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:43213 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.12.126:43213: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
I20260805 23:05:45.229291 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:45.219274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.3.12.126:43213)
0805 23:05:45.219515 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:45.219520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:45.219534 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:45.219563 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:45.219566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:45.219606 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:45.219709 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:45.220741 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.221304 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.222349 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.222622 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.222703 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.224366 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:05:45.224373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:05:45.224392 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:05:45.224424 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:05:45.226832 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:45.227326 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:45.227332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:45.227339 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:45.227488 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:05:45.227751 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:05:45.227754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:05:45.227756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:05:45.228149 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:05:45.228396 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0805 23:05:45.228656 (+ 260us) server_negotiation.cc:299] Negotiation successful
0805 23:05:45.229113 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260805 23:05:45.231573 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:45.225662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:60073 (local address 127.3.12.126:43213)
0805 23:05:45.225787 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:45.225791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:45.225830 (+ 39us) server_negotiation.cc:407] Connection header received
0805 23:05:45.225859 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:45.225861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:45.225893 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:45.225948 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:45.226405 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.226929 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.227903 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.228120 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.228198 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.231446 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:60073: BlockingRecv error: recv got EOF from 127.3.12.67:60073 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260805 23:05:45.231581 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:45.225566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:60073)
0805 23:05:45.225699 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:45.225715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:45.225789 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:45.226006 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:45.226011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:45.226167 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:45.226287 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.226298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.227047 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.227050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.227783 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.227796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.228236 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.228243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.228368 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.228880 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:45.228898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:45.231330 (+ 2432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:45.239007 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:45.232412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:41789)
0805 23:05:45.232534 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:45.232550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:45.232633 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:45.232914 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:45.232917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:45.233094 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:45.233202 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.233211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.234364 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.234369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.235126 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.235138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.235888 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.235892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.235982 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.237198 (+ 1216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:45.237213 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:45.238916 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260805 23:05:45.239142 16183 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:45.232448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:41789 (local address 127.3.12.126:43213)
0805 23:05:45.232615 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:45.232619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:45.232637 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:05:45.232728 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:45.232731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:45.232788 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:45.232885 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:45.233338 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.234223 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.235272 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.235665 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.235770 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.239038 (+ 3268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:41789: BlockingRecv error: recv got EOF from 127.3.12.67:41789 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260805 23:05:45.239434 16572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:45 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:45.846081 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:45.840209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:36751)
0805 23:05:45.840587 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:45.840601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:45.840710 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:45.840958 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:45.840960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:45.841175 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:45.841281 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.841287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.842067 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.842070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.842746 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:45.842756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.843227 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.843230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:45.843296 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.843903 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:45.843916 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:45.845942 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":86,"threads_started":1}
W20260805 23:05:45.846326 16587 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:45.840283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:36751 (local address 127.3.12.126:43213)
0805 23:05:45.840596 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:45.840600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:45.840672 (+ 72us) server_negotiation.cc:407] Connection header received
0805 23:05:45.840791 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:45.840793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:45.840836 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:45.840914 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:45.841394 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.841962 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.842865 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:45.843109 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:45.843196 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:45.846116 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:36751: BlockingRecv error: recv got EOF from 127.3.12.65:36751 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":132,"threads_started":1}
Aug 05 23:05:46 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:46 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:46.046902 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:46.041172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:58987)
0805 23:05:46.041468 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:46.041486 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:46.041614 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:46.041892 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:46.041895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:46.042116 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:46.042218 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:46.042224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.043014 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.043018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:46.043601 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:46.043610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.044159 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.044163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:46.044283 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:46.044806 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:46.044819 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:46.046768 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
W20260805 23:05:46.047119 16587 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:46.041241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:58987 (local address 127.3.12.126:43213)
0805 23:05:46.041395 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:46.041402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:46.041549 (+ 147us) server_negotiation.cc:407] Connection header received
0805 23:05:46.041725 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:46.041727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:46.041781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:46.041867 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:46.042347 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.042891 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.043776 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.044038 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.044153 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:46.046983 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:58987: BlockingRecv error: recv got EOF from 127.3.12.66:58987 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 05 23:05:46 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:46 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:46.246304 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:46.240145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:45023)
0805 23:05:46.240498 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:46.240512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:46.240622 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:46.240857 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:46.240860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:46.241115 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:46.241221 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:46.241227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.242015 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.242018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:46.242665 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:46.242673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.243222 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.243226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:46.243332 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:46.243958 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:46.243975 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:46.246098 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":167,"threads_started":1}
W20260805 23:05:46.246343 16587 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:46.240302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:45023 (local address 127.3.12.126:43213)
0805 23:05:46.240444 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:46.240449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:46.240568 (+ 119us) server_negotiation.cc:407] Connection header received
0805 23:05:46.240700 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:46.240702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:46.240750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:46.240819 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:46.241333 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.241872 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.242806 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.243075 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.243197 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:46.246228 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:45023: BlockingRecv error: recv got EOF from 127.3.12.67:45023 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 05 23:05:46 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:46 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:46.853245 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:46.846895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:39319)
0805 23:05:46.847238 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:46.847252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:46.847348 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:46.847613 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:46.847616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:46.847807 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:46.847910 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:46.847916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.849034 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.849038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:46.849734 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:46.849760 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.850289 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.850293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:46.850364 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:46.850921 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:46.850935 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:46.853094 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
W20260805 23:05:46.853389 16591 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:46.847043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:39319 (local address 127.3.12.126:43213)
0805 23:05:46.847301 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:46.847304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:46.847318 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:46.847435 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:46.847437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:46.847479 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:46.847569 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:46.848053 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.848840 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.849912 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:46.850151 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:46.850248 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:46.853250 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:39319: BlockingRecv error: recv got EOF from 127.3.12.65:39319 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
Aug 05 23:05:47 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:47 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:47.053946 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:47.047788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:52171)
0805 23:05:47.048213 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:47.048227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:47.048407 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:47.048736 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:47.048739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:47.048960 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:47.049062 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:47.049068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.049875 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.049879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:47.050444 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:47.050452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.050963 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.050967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:47.051063 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:47.051736 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:47.051749 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:47.053782 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":136,"threads_started":1}
W20260805 23:05:47.054026 16591 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:47.047953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:52171 (local address 127.3.12.126:43213)
0805 23:05:47.048144 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:47.048149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:47.048335 (+ 186us) server_negotiation.cc:407] Connection header received
0805 23:05:47.048528 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:47.048531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:47.048611 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:47.048704 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:47.049195 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.049756 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.050578 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.050823 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.050938 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:47.053930 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:52171: BlockingRecv error: recv got EOF from 127.3.12.66:52171 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 05 23:05:47 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:47 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:47.252940 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:47.247235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:57355)
0805 23:05:47.247525 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:47.247546 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:47.247655 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:47.247891 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:47.247894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:47.248127 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:47.248231 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:47.248237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.249032 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.249035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:47.249644 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:47.249653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.250109 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.250112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:47.250180 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:47.250677 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:47.250690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:47.252786 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
W20260805 23:05:47.253101 16591 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:47.247399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:57355 (local address 127.3.12.126:43213)
0805 23:05:47.247559 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:47.247562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:47.247598 (+ 36us) server_negotiation.cc:407] Connection header received
0805 23:05:47.247732 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:47.247734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:47.247780 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:47.247848 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:47.248386 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.248907 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.249753 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.249971 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.250085 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:47.252923 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:57355: BlockingRecv error: recv got EOF from 127.3.12.67:57355 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 05 23:05:47 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:47 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:47.859977 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:47.854136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:43635)
0805 23:05:47.854444 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:47.854461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:47.854568 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:47.854835 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:47.854838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:47.855047 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:47.855153 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:47.855159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.855974 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.855976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:47.856588 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:47.856597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.857189 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.857193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:47.857256 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:47.857845 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:47.857858 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:47.859833 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":132,"threads_started":1}
W20260805 23:05:47.860194 16597 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:47.854337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:43635 (local address 127.3.12.126:43213)
0805 23:05:47.854574 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:47.854577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:47.854593 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:05:47.854635 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:47.854637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:47.854678 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:47.854763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:47.855266 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.855867 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.856697 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:47.857056 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:47.857186 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:47.860021 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:43635: BlockingRecv error: recv got EOF from 127.3.12.65:43635 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
Aug 05 23:05:48 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:48 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:48.061045 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:48.054835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:55741)
0805 23:05:48.055122 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:48.055138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:48.055251 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:48.055613 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:48.055619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:48.055836 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:48.055957 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:48.055966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.056872 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.056875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:48.057492 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:48.057509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.057970 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.057972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:48.058039 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:48.058571 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:48.058584 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:48.060897 (+ 2313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
W20260805 23:05:48.061318 16597 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:48.055044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:55741 (local address 127.3.12.126:43213)
0805 23:05:48.055194 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:48.055197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:48.055212 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:48.055347 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:48.055350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:48.055397 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:48.055474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:48.056106 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.056755 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.057611 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.057835 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.057944 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:48.061086 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:55741: BlockingRecv error: recv got EOF from 127.3.12.66:55741 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 05 23:05:48 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:48 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:48.261400 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:48.253859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:55145)
0805 23:05:48.254255 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:48.254272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:48.254393 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:48.254703 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:48.254706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:48.254953 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:48.255111 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:48.255123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.256420 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.256426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:48.257501 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:48.257515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.258373 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.258376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:48.258440 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:48.259034 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:48.259051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:48.261260 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":91,"threads_started":1}
W20260805 23:05:48.261583 16597 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:48.253902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:55145 (local address 127.3.12.126:43213)
0805 23:05:48.254078 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:48.254082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:48.254339 (+ 257us) server_negotiation.cc:407] Connection header received
0805 23:05:48.254522 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:48.254525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:48.254594 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:48.254674 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:48.255266 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.256267 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.257654 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.258105 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.258200 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:48.261413 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:55145: BlockingRecv error: recv got EOF from 127.3.12.67:55145 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 05 23:05:48 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:48 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:48.872049 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:48.861174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:53673)
0805 23:05:48.861466 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:48.861480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:48.861589 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:48.861889 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:48.861891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:48.862082 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:48.862183 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:48.862189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.863058 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.863061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:48.863627 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:48.863636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.864153 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.864156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:48.864226 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:48.869878 (+ 5652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:48.869891 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:48.871921 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
W20260805 23:05:48.872341 16602 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:48.861274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:53673 (local address 127.3.12.126:43213)
0805 23:05:48.861536 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:48.861540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:48.861555 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:48.861713 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:48.861716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:48.861767 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:48.861857 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:48.862319 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.862941 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.863763 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:48.864036 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:48.864132 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:48.872174 (+ 8042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:53673: BlockingRecv error: recv got EOF from 127.3.12.65:53673 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
Aug 05 23:05:49 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:49 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:49.068184 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:49.062090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:36625)
0805 23:05:49.062442 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:49.062457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:49.062612 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:49.062934 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:49.062937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:49.063135 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:49.063247 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:49.063253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.064099 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.064102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:49.064730 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:49.064740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.065244 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.065247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:49.065320 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:49.065946 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:49.065999 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:49.068062 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
W20260805 23:05:49.068426 16602 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:49.062192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:36625 (local address 127.3.12.126:43213)
0805 23:05:49.062390 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:49.062397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:49.062569 (+ 172us) server_negotiation.cc:407] Connection header received
0805 23:05:49.062719 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:49.062722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:49.062771 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:49.062842 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:49.063402 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.063944 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.064845 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.065099 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.065185 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:49.068270 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:36625: BlockingRecv error: recv got EOF from 127.3.12.66:36625 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 05 23:05:49 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:49 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:49.268576 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:49.262290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:34743)
0805 23:05:49.262681 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:49.262694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:49.262835 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:49.263115 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:49.263118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:49.263347 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:49.263452 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:49.263458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.264300 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.264304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:49.265085 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:49.265098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.265719 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.265729 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:49.265858 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:49.266382 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:49.266394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:49.268440 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":121,"threads_started":1}
W20260805 23:05:49.268802 16602 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:49.262371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:34743 (local address 127.3.12.126:43213)
0805 23:05:49.262546 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:49.262553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:49.262775 (+ 222us) server_negotiation.cc:407] Connection header received
0805 23:05:49.262947 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:49.262952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:49.263005 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:49.263094 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:49.263591 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.264141 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.265253 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.265598 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.265704 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:49.268670 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:34743: BlockingRecv error: recv got EOF from 127.3.12.67:34743 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 05 23:05:49 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:49 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:49.878880 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:49.873043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:57441)
0805 23:05:49.873366 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:49.873389 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:49.873537 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:49.873787 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:49.873789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:49.873990 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:49.874099 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:49.874104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.874947 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.874952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:49.875616 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:49.875626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.876131 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.876134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:49.876209 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:49.876788 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:49.876801 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:49.878758 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
W20260805 23:05:49.879106 16606 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:49.873084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:57441 (local address 127.3.12.126:43213)
0805 23:05:49.873349 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:49.873354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:49.873484 (+ 130us) server_negotiation.cc:407] Connection header received
0805 23:05:49.873626 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:49.873628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:49.873674 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:49.873745 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:49.874227 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.874807 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.875765 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:49.876006 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:49.876100 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:49.878939 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:57441: BlockingRecv error: recv got EOF from 127.3.12.65:57441 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
Aug 05 23:05:50 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:50 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:50.074959 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:50.069217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:39131)
0805 23:05:50.069527 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:50.069541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:50.069638 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:50.069889 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:50.069891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:50.070081 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:50.070178 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:50.070184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.070941 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.070947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:50.071571 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:50.071581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.072069 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.072074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:50.072151 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:50.072753 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:50.072773 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:50.074844 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
W20260805 23:05:50.075137 16606 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:50.069371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:39131 (local address 127.3.12.126:43213)
0805 23:05:50.069535 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:50.069538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:50.069614 (+ 76us) server_negotiation.cc:407] Connection header received
0805 23:05:50.069721 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:50.069723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:50.069766 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:50.069851 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:50.070280 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.070816 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.071706 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.071934 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.072049 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:50.074977 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:39131: BlockingRecv error: recv got EOF from 127.3.12.66:39131 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 05 23:05:50 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:50 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:50.275346 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:50.269380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:38883)
0805 23:05:50.269735 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:50.269748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:50.269883 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:50.270166 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:50.270169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:50.270391 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:50.270481 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:50.270487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.271312 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.271315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:50.271948 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:50.271957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.272559 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.272562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:50.272658 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:50.273221 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:50.273233 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:50.275222 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
W20260805 23:05:50.275681 16606 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:50.269513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:38883 (local address 127.3.12.126:43213)
0805 23:05:50.269687 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:50.269692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:50.269823 (+ 131us) server_negotiation.cc:407] Connection header received
0805 23:05:50.269992 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:50.269995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:50.270054 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:50.270140 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:50.270608 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.271143 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.272102 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.272418 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.272537 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:50.275399 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:38883: BlockingRecv error: recv got EOF from 127.3.12.67:38883 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 05 23:05:50 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:50 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:50.885864 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:50.879707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:36981)
0805 23:05:50.879993 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:50.880009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:50.880107 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:50.880397 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:50.880400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:50.880674 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:50.880779 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:50.880786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.881572 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.881577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:50.882176 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:50.882185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.882717 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.882720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:50.882786 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:50.883492 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:50.883540 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:50.885679 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
W20260805 23:05:50.885967 16613 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:50.879811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:36981 (local address 127.3.12.126:43213)
0805 23:05:50.880081 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:50.880085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:50.880100 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:50.880203 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:50.880205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:50.880270 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:50.880351 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:50.880894 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.881437 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.882317 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:50.882581 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:50.882708 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:50.885823 (+ 3115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:36981: BlockingRecv error: recv got EOF from 127.3.12.65:36981 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
Aug 05 23:05:51 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:51 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:51.082038 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:51.075770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:36473)
0805 23:05:51.076080 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:51.076095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:51.076204 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:51.076560 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:51.076562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:51.076758 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:51.076856 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:51.076861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.077696 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.077698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:51.078259 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:51.078267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.078753 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.078756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:51.078822 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:51.079421 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:51.079434 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:51.081858 (+ 2424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
W20260805 23:05:51.082222 16613 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:51.075866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:36473 (local address 127.3.12.126:43213)
0805 23:05:51.076059 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:51.076066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:51.076306 (+ 240us) server_negotiation.cc:407] Connection header received
0805 23:05:51.076375 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:51.076377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:51.076431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:51.076521 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:51.077017 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.077594 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.078377 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.078639 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.078729 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:51.082076 (+ 3347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:36473: BlockingRecv error: recv got EOF from 127.3.12.66:36473 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 05 23:05:51 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:51 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:51.282186 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:51.276332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:42425)
0805 23:05:51.276625 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:51.276638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:51.276778 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:51.277098 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:51.277102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:51.277317 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:51.277409 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:51.277415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.278202 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.278207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:51.278769 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:51.278778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.279247 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.279251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:51.279337 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:51.279935 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:51.279947 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:51.282000 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":92,"threads_started":1}
W20260805 23:05:51.282226 16613 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:51.276359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:42425 (local address 127.3.12.126:43213)
0805 23:05:51.276527 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:51.276531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:51.276715 (+ 184us) server_negotiation.cc:407] Connection header received
0805 23:05:51.276881 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:51.276884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:51.276941 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:51.277029 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:51.277556 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.278087 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.278892 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.279125 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.279241 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:51.282126 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:42425: BlockingRecv error: recv got EOF from 127.3.12.67:42425 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 05 23:05:51 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:51 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:51.896391 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:51.886854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:35115)
0805 23:05:51.887371 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:51.887413 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:51.887710 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:51.888790 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:51.888794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:51.889122 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:51.889280 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:51.889290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.890427 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.890430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:51.891059 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:51.891068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.891639 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.891641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:51.891808 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:51.892895 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:51.892909 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:51.896198 (+ 3289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":134,"threads_started":1}
W20260805 23:05:51.896975 16620 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:51.886972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:35115 (local address 127.3.12.126:43213)
0805 23:05:51.887458 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:51.887468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:51.887591 (+ 123us) server_negotiation.cc:407] Connection header received
0805 23:05:51.888088 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:51.888119 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:51.888330 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:51.888774 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:51.889440 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.890307 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.891224 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:51.891531 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:51.891625 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:51.896430 (+ 4805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:35115: BlockingRecv error: recv got EOF from 127.3.12.65:35115 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":87,"threads_started":1}
Aug 05 23:05:52 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:52 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:52.095664 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:52.083287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:53793)
0805 23:05:52.084087 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:52.084121 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:52.084564 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:52.085593 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:52.085608 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:52.086706 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:52.088059 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:52.088079 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.089665 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.089670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:52.090498 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:52.090510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.091078 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.091102 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:52.091359 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:52.092599 (+ 1240us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:52.092612 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:52.095344 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":154,"threads_started":1}
W20260805 23:05:52.095984 16620 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:52.083584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:53793 (local address 127.3.12.126:43213)
0805 23:05:52.083759 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:52.083763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:52.084430 (+ 667us) server_negotiation.cc:407] Connection header received
0805 23:05:52.084989 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:52.084999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:52.085117 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:52.085367 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:52.088631 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.089536 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.090632 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.090857 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.090943 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:52.095750 (+ 4807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:53793: BlockingRecv error: recv got EOF from 127.3.12.66:53793 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 05 23:05:52 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:52 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:52.293434 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:52.283193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:47795)
0805 23:05:52.284176 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:52.284246 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:52.284924 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:52.285994 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:52.285997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:52.286243 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:52.286337 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:52.286344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.287261 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.287268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:52.289161 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:52.289197 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.290136 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.290140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:52.290227 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:52.290875 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:52.290891 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:52.293301 (+ 2410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":121,"threads_started":1}
W20260805 23:05:52.293704 16620 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:52.283294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:47795 (local address 127.3.12.126:43213)
0805 23:05:52.283597 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:52.283627 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:52.284647 (+ 1020us) server_negotiation.cc:407] Connection header received
0805 23:05:52.285597 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:52.285606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:52.285839 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:52.285966 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:52.286467 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.287149 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.289630 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.290013 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.290130 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:52.293505 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:47795: BlockingRecv error: recv got EOF from 127.3.12.67:47795 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 05 23:05:52 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:52 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:52.906234 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:52.897640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:51129)
0805 23:05:52.898092 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:52.898108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:52.898232 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:52.898747 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:52.898750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:52.899183 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:52.899316 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:52.899323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.900978 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.900982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:52.901677 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:52.901689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.902423 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.902428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:52.902520 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:52.903066 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:52.903079 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:52.906051 (+ 2972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
W20260805 23:05:52.906316 16625 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:52.897866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:51129 (local address 127.3.12.126:43213)
0805 23:05:52.898133 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:52.898137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:52.898196 (+ 59us) server_negotiation.cc:407] Connection header received
0805 23:05:52.898337 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:52.898343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:52.898437 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:52.898686 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:52.899530 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.900831 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.901930 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:52.902281 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:52.902416 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:52.906187 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:51129: BlockingRecv error: recv got EOF from 127.3.12.65:51129 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
Aug 05 23:05:53 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:53 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:53.105983 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:53.097480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:43313)
0805 23:05:53.098067 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:53.098081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:53.098216 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:53.098786 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:53.098789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:53.099155 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:53.099273 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:53.099280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.100722 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.100725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:53.101470 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:53.101480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.101949 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.101951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:53.102015 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:53.102566 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:53.102578 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:53.105735 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":201,"threads_started":1}
W20260805 23:05:53.106068 16625 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:53.097634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:43313 (local address 127.3.12.126:43213)
0805 23:05:53.098054 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:53.098059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:53.098146 (+ 87us) server_negotiation.cc:407] Connection header received
0805 23:05:53.098511 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:53.098533 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:53.098595 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:53.098741 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:53.099514 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.100612 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.101606 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.101848 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.101936 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:53.105927 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:43313: BlockingRecv error: recv got EOF from 127.3.12.66:43313 (error 108)
Metrics: {"server-negotiator.queue_time_us":200}
Aug 05 23:05:53 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:53 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:53.301576 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:53.294583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:50819)
0805 23:05:53.295067 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:53.295085 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:53.295264 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:53.295858 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:53.295861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:53.296093 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:53.296220 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:53.296226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.297167 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.297174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:53.297902 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:53.297933 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.298485 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.298488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:53.298579 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:53.299200 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:53.299213 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:53.301395 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":135,"threads_started":1}
W20260805 23:05:53.301682 16625 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:53.294739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:50819 (local address 127.3.12.126:43213)
0805 23:05:53.295051 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:53.295056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:53.295190 (+ 134us) server_negotiation.cc:407] Connection header received
0805 23:05:53.295653 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:53.295673 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:53.295734 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:53.295831 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:53.296411 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.296974 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.298088 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.298356 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.298478 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:53.301544 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:50819: BlockingRecv error: recv got EOF from 127.3.12.67:50819 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Aug 05 23:05:53 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:53 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:53.922585 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:53.907774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:45321)
0805 23:05:53.908465 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:53.908532 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:53.908753 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:53.909245 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:53.909247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:53.909436 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:53.909555 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:53.909561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.910729 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.910740 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:53.911852 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:53.911892 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.912982 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.912993 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:53.913219 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:53.919829 (+ 6610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:53.919843 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:53.922391 (+ 2548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":272,"threads_started":1}
W20260805 23:05:53.922659 16630 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:53.908699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:45321 (local address 127.3.12.126:43213)
0805 23:05:53.909012 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:53.909016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:53.909027 (+ 11us) server_negotiation.cc:407] Connection header received
0805 23:05:53.909062 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:53.909064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:53.909116 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:53.909194 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:53.909673 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.910481 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.912130 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:53.912739 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:53.912958 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:53.922534 (+ 9576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:45321: BlockingRecv error: recv got EOF from 127.3.12.65:45321 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":130,"threads_started":1}
Aug 05 23:05:54 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:54 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:54.119309 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:54.107922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:43501)
0805 23:05:54.108708 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:54.108731 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:54.108981 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:54.109364 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:54.109367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:54.109689 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:54.109916 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:54.109931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.111136 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.111139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:54.111697 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:54.111705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.112426 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.112430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:54.112530 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:54.114411 (+ 1881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:54.114525 (+ 114us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:54.119173 (+ 4648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":377,"threads_started":1}
W20260805 23:05:54.119583 16630 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:54.108059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:43501 (local address 127.3.12.126:43213)
0805 23:05:54.108554 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:54.108577 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:54.108891 (+ 314us) server_negotiation.cc:407] Connection header received
0805 23:05:54.109095 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:54.109098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:54.109171 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:54.109324 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:54.110208 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.111033 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.111809 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.112299 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.112406 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:54.119364 (+ 6958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:43501: BlockingRecv error: recv got EOF from 127.3.12.66:43501 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Aug 05 23:05:54 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:54 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:54.308699 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:54.302636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:33393)
0805 23:05:54.302967 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:54.302983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:54.303174 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:54.303527 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:54.303530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:54.303816 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:54.303956 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:54.303966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.304789 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.304792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:54.305390 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:54.305399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.305856 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.305859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:54.305929 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:54.306569 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:54.306591 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:54.308580 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":114,"threads_started":1}
W20260805 23:05:54.308894 16630 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:54.302714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:33393 (local address 127.3.12.126:43213)
0805 23:05:54.302884 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:54.302889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:54.303115 (+ 226us) server_negotiation.cc:407] Connection header received
0805 23:05:54.303314 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:54.303317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:54.303377 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:54.303468 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:54.304097 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.304692 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.305508 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.305726 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.305807 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:54.308727 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:33393: BlockingRecv error: recv got EOF from 127.3.12.67:33393 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 05 23:05:54 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:54 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:54.929216 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:54.923435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:42245)
0805 23:05:54.923727 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:54.923745 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:54.923854 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:54.924075 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:54.924077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:54.924358 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:54.924468 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:54.924474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.925226 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.925228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:54.925810 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:54.925818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.926255 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.926258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:54.926320 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:54.926835 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:54.926846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:54.928999 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
W20260805 23:05:54.929466 16639 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:54.923594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:42245 (local address 127.3.12.126:43213)
0805 23:05:54.923859 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:54.923863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:54.923877 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:54.923932 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:54.923935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:54.923975 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:54.924057 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:54.924585 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.925124 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.925914 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:54.926151 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:54.926232 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:54.929273 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:42245: BlockingRecv error: recv got EOF from 127.3.12.65:42245 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
Aug 05 23:05:55 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:55 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:55.126111 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:55.120410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:42859)
0805 23:05:55.120733 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:55.120749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:55.120872 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:55.121210 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:55.121212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:55.121394 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:55.121489 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:55.121494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.122279 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.122282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:55.122844 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:55.122851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.123276 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.123278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:55.123337 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:55.123882 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:55.123894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:55.125944 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
W20260805 23:05:55.126246 16639 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:55.120449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:42859 (local address 127.3.12.126:43213)
0805 23:05:55.120614 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:55.120618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:55.120846 (+ 228us) server_negotiation.cc:407] Connection header received
0805 23:05:55.121002 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:55.121008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:55.121068 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:55.121170 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:55.121600 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.122169 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.122957 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.123175 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.123257 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:55.126125 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:42859: BlockingRecv error: recv got EOF from 127.3.12.66:42859 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 05 23:05:55 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:55 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:55.315481 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:55.309564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:40433)
0805 23:05:55.309856 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:55.309870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:55.309960 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:55.310204 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:55.310206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:55.310397 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:55.310499 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:55.310505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.311280 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.311283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:55.311863 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:55.311872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.312383 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.312386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:55.312453 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:55.313052 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:55.313064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:55.315295 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
W20260805 23:05:55.315604 16639 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:55.309664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:40433 (local address 127.3.12.126:43213)
0805 23:05:55.309848 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:55.309852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:55.309939 (+ 87us) server_negotiation.cc:407] Connection header received
0805 23:05:55.310044 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:55.310046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:55.310093 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:55.310168 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:55.310607 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.311158 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.312046 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.312286 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.312363 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:55.315463 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:40433: BlockingRecv error: recv got EOF from 127.3.12.67:40433 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 05 23:05:55 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:55 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:55.936026 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:55.930222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:51379)
0805 23:05:55.930524 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:55.930539 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:55.930634 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:55.930943 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:55.930945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:55.931137 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:55.931242 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:55.931248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.932077 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.932080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:55.932761 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:55.932773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.933246 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.933249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:55.933315 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:55.933842 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:55.933855 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:55.935886 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
W20260805 23:05:55.936357 16645 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:55.930292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:51379 (local address 127.3.12.126:43213)
0805 23:05:55.930569 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:55.930573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:55.930588 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:55.930728 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:55.930730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:55.930774 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:55.930861 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:55.931372 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.931933 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.932877 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:55.933136 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:55.933225 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:55.936075 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:51379: BlockingRecv error: recv got EOF from 127.3.12.65:51379 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":113,"threads_started":1}
Aug 05 23:05:56 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:56 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:56.133087 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:56.127157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:34359)
0805 23:05:56.127462 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:56.127478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:56.127611 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:56.127934 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:56.127937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:56.128205 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:56.128380 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:56.128386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.129198 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.129200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:56.129749 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:56.129756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.130253 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.130256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:56.130318 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:56.130860 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:56.130872 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:56.132916 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
W20260805 23:05:56.133196 16645 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:56.127220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:34359 (local address 127.3.12.126:43213)
0805 23:05:56.127379 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:56.127384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:56.127595 (+ 211us) server_negotiation.cc:407] Connection header received
0805 23:05:56.127720 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:56.127724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:56.127786 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:56.127895 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:56.128507 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.129092 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.129873 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.130140 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.130238 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:56.133065 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:34359: BlockingRecv error: recv got EOF from 127.3.12.66:34359 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 05 23:05:56 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:56 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:56.323046 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:56.316577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:46265)
0805 23:05:56.316865 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:56.316880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:56.316980 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:56.317242 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:56.317244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:56.317433 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:56.317534 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:56.317539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.318600 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.318603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:56.319515 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:56.319531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.320165 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.320169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:56.320306 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:56.320987 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:56.321013 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:56.322893 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
W20260805 23:05:56.323130 16645 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:56.316628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:46265 (local address 127.3.12.126:43213)
0805 23:05:56.316753 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:56.316757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:56.316955 (+ 198us) server_negotiation.cc:407] Connection header received
0805 23:05:56.317087 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:56.317090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:56.317138 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:56.317198 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:56.317636 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.318450 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.319656 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.320038 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.320148 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:56.323037 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:46265: BlockingRecv error: recv got EOF from 127.3.12.67:46265 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 05 23:05:56 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:56 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:56.942932 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:56.936930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:33317)
0805 23:05:56.937231 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:56.937251 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:56.937353 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:56.937601 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:56.937603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:56.937793 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:56.937894 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:56.937900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.938730 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.938733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:56.939322 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:56.939332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.939899 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.939902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:56.939997 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:56.940652 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:56.940664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:56.942806 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":115,"threads_started":1}
W20260805 23:05:56.943177 16649 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:56.936934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:33317 (local address 127.3.12.126:43213)
0805 23:05:56.937225 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:56.937228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:56.937305 (+ 77us) server_negotiation.cc:407] Connection header received
0805 23:05:56.937419 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:56.937421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:56.937463 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:56.937560 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:56.938032 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.938595 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.939482 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:56.939757 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:56.939860 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:56.942980 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:33317: BlockingRecv error: recv got EOF from 127.3.12.65:33317 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
Aug 05 23:05:57 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:57 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:57.139806 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:57.133862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:41359)
0805 23:05:57.134245 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:57.134269 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:57.134415 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:57.134697 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:57.134700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:57.134923 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:57.135058 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:57.135065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.135857 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.135862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:57.136506 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:57.136515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.136992 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.137002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:57.137098 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:57.137604 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:57.137616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:57.139626 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":152,"threads_started":1}
W20260805 23:05:57.139864 16649 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:57.133992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:41359 (local address 127.3.12.126:43213)
0805 23:05:57.134212 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:57.134217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:57.134349 (+ 132us) server_negotiation.cc:407] Connection header received
0805 23:05:57.134525 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:57.134528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:57.134576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:57.134674 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:57.135187 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.135742 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.136624 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.136864 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.136986 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:57.139755 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:41359: BlockingRecv error: recv got EOF from 127.3.12.66:41359 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 05 23:05:57 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:57 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:57.330379 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:57.323994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:43635)
0805 23:05:57.324628 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:57.324643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:57.324780 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:57.325137 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:57.325140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:57.325327 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:57.325428 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:57.325434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.326191 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.326193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:57.326761 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:57.326770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.327217 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.327220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:57.327287 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:57.327874 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:57.327886 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:57.330193 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":107,"threads_started":1}
W20260805 23:05:57.330988 16649 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:57.324094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:43635 (local address 127.3.12.126:43213)
0805 23:05:57.324229 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:57.324233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:57.324711 (+ 478us) server_negotiation.cc:407] Connection header received
0805 23:05:57.324912 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:57.324916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:57.324983 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:57.325085 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:57.325535 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.326091 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.326861 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.327099 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.327177 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:57.330407 (+ 3230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:43635: BlockingRecv error: recv got EOF from 127.3.12.67:43635 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 05 23:05:57 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:57 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:57.950244 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:57.944020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:37625)
0805 23:05:57.944347 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:57.944361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:57.944480 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:57.944850 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:57.944854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:57.945094 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:57.945244 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:57.945252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.946204 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.946207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:57.946772 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:57.946780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.947240 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.947243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:57.947311 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:57.947847 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:57.947859 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:57.950119 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":114,"threads_started":1}
W20260805 23:05:57.950397 16654 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:57.944104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:37625 (local address 127.3.12.126:43213)
0805 23:05:57.944406 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:57.944411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:57.944425 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:57.944606 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:57.944608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:57.944690 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:57.944797 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:57.945384 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.946075 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.946884 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:57.947128 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:57.947213 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:57.950286 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:37625: BlockingRecv error: recv got EOF from 127.3.12.65:37625 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
Aug 05 23:05:58 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:58 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:58.146984 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:58.140878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:34451)
0805 23:05:58.141188 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:58.141203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:58.141301 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:58.141661 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:58.141664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:58.141870 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:58.142006 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:58.142012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.142867 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.142870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:58.143480 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:58.143491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.144032 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.144036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:58.144125 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:58.144737 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:58.144751 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:58.146810 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":110,"threads_started":1}
W20260805 23:05:58.147177 16654 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:58.141127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:34451 (local address 127.3.12.126:43213)
0805 23:05:58.141274 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:58.141278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:58.141292 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:58.141394 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:58.141396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:58.141443 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:58.141529 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:58.142179 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.142740 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.143612 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.143846 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.143996 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:58.147002 (+ 3006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:34451: BlockingRecv error: recv got EOF from 127.3.12.66:34451 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 05 23:05:58 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:58 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:58.337143 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:58.331370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:59401)
0805 23:05:58.331839 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:58.331851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:58.332017 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:58.332294 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:58.332303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:58.332491 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:58.332590 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:58.332596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.333328 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.333331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:58.333894 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:58.333902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.334360 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.334363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:58.334437 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:58.334940 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:58.334990 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:58.337005 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":134,"threads_started":1}
W20260805 23:05:58.337337 16654 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:58.331484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:59401 (local address 127.3.12.126:43213)
0805 23:05:58.331630 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:58.331636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:58.331949 (+ 313us) server_negotiation.cc:407] Connection header received
0805 23:05:58.332105 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:58.332107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:58.332156 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:58.332218 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:58.332696 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.333228 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.333999 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.334221 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.334300 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:58.337197 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:59401: BlockingRecv error: recv got EOF from 127.3.12.67:59401 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 05 23:05:58 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:58 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:58.962239 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:58.951291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:56909)
0805 23:05:58.951618 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:58.951633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:58.951732 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:58.952009 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:58.952012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:58.952205 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:58.952338 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:58.952344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.953230 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.953236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:58.954065 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:58.954076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.954589 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.954591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:58.954655 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:58.959969 (+ 5314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:58.959984 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:58.962102 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":120,"threads_started":1}
W20260805 23:05:58.962469 16659 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:58.951384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:56909 (local address 127.3.12.126:43213)
0805 23:05:58.951758 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:58.951762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:58.951777 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:05:58.951819 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:58.951821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:58.951863 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:58.951965 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:58.952451 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.953109 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.954205 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:58.954477 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:58.954565 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:58.962303 (+ 7738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:56909: BlockingRecv error: recv got EOF from 127.3.12.65:56909 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
Aug 05 23:05:59 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:59 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:59.153580 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:59.147954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:46327)
0805 23:05:59.148327 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:59.148341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:59.148446 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:59.148708 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:59.148711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:59.148905 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:59.149006 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:59.149011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.149761 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.149764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:59.150337 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:59.150347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.150807 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.150810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:59.150871 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:59.151444 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:59.151457 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:59.153428 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":137,"threads_started":1}
W20260805 23:05:59.153745 16659 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:59.148045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:46327 (local address 127.3.12.126:43213)
0805 23:05:59.148347 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:59.148351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:59.148405 (+ 54us) server_negotiation.cc:407] Connection header received
0805 23:05:59.148543 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:59.148545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:59.148590 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:59.148668 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:59.149111 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.149649 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.150457 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.150702 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.150785 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:59.153580 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:46327: BlockingRecv error: recv got EOF from 127.3.12.66:46327 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Aug 05 23:05:59 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:59 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.67@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:59.343487 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:59.337926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.67:39179)
0805 23:05:59.338212 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:59.338228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:59.338336 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:59.338558 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:59.338561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:59.338745 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:59.338841 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:59.338846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.339597 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.339600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:59.340210 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:59.340219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.340679 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.340682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:59.340743 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:59.341319 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:59.341332 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:59.343303 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
W20260805 23:05:59.343654 16659 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:59.338032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:39179 (local address 127.3.12.126:43213)
0805 23:05:59.338180 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:59.338184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:59.338286 (+ 102us) server_negotiation.cc:407] Connection header received
0805 23:05:59.338401 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:59.338403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:59.338448 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:59.338522 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:59.338938 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.339483 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.340361 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.340582 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.340659 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:59.343493 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.67:39179: BlockingRecv error: recv got EOF from 127.3.12.67:39179 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 05 23:05:59 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:05:59 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.65@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:05:59.968750 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:05:59.963294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.65:45509)
0805 23:05:59.963586 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0805 23:05:59.963600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:05:59.963683 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:05:59.963933 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:05:59.963935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:05:59.964137 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:05:59.964231 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:59.964237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.965085 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.965088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:59.965657 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:05:59.965665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.966114 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.966116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:05:59.966180 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:59.966679 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:05:59.966690 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:05:59.968622 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
W20260805 23:05:59.968894 16665 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:05:59.963367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:45509 (local address 127.3.12.126:43213)
0805 23:05:59.963652 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0805 23:05:59.963656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:05:59.963670 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:05:59.963774 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:05:59.963776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:05:59.963819 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:05:59.963898 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:05:59.964405 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.964986 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.965759 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:05:59.966017 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:05:59.966097 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:05:59.968761 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.65:45509: BlockingRecv error: recv got EOF from 127.3.12.65:45509 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":100,"threads_started":1}
Aug 05 23:06:00 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
Aug 05 23:06:00 dist-test-slave-2d19 krb5kdc[15100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.12.66@KRBTEST.COM for oryx/127.3.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260805 23:06:00.160907 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:00.154591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43213 (local address 127.3.12.66:43685)
0805 23:06:00.154872 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:00.154887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:00.155000 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:00.155247 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:00.155249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:00.155440 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:00.155552 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:00.155558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.156436 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.156439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:00.157209 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:00.157221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.157674 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.157677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:00.157739 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:00.158496 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:06:00.158522 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:00.160729 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.12.126:43213: Server oryx/127.3.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
W20260805 23:06:00.160969 16665 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:06:00.154692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:43685 (local address 127.3.12.126:43213)
0805 23:06:00.154845 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:00.154848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:00.155000 (+ 152us) server_negotiation.cc:407] Connection header received
0805 23:06:00.155077 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:00.155080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:00.155129 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:00.155208 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:00.155656 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.156332 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.157320 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.157542 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.157672 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:00.160861 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.12.66:43685: BlockingRecv error: recv got EOF from 127.3.12.66:43685 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:06:00.227543 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 15960
I20260805 23:06:00.240800 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 16186
I20260805 23:06:00.260627 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 16380
I20260805 23:06:00.271943 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 15851
2026-08-05T23:06:00Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18385 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-05T23:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:06:00Z Disabled control of system clock
I20260805 23:06:00.310129 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:45219
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260805 23:06:00.430325 16671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:00.430589 16671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:00.430624 16671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:00.434628 16671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:06:00.434685 16671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:00.434701 16671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:00.434718 16671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:06:00.434734 16671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:06:00.439527 16671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:45219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:00.440609 16671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:00.442000 16671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:06:00.451385 16671 server_base.cc:1034] running on GCE node
W20260805 23:06:00.451341 16687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:00.451341 16682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:00.451399 16685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:00.452154 16671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:00.453267 16671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:00.454442 16671 hybrid_clock.cc:648] HybridClock initialized: now 1785971160454427 us; error 26 us; skew 500 ppm
I20260805 23:06:00.456439 16671 webserver.cc:460] Webserver started at http://127.3.12.126:46349/ using document root <none> and password file <none>
I20260805 23:06:00.457027 16671 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:00.457090 16671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:00.457337 16671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:00.459457 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "7dddcc49085e43cab42be18f9085bd23"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "e2b08cfc0a1e3cfffed6f79ae88311db"
server_key_iv: "b055ba380214de0daea54adf82cf72ff"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.460026 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "7dddcc49085e43cab42be18f9085bd23"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "e2b08cfc0a1e3cfffed6f79ae88311db"
server_key_iv: "b055ba380214de0daea54adf82cf72ff"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.463649 16671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:06:00.466230 16694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:00.467487 16671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:06:00.467638 16671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "7dddcc49085e43cab42be18f9085bd23"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "e2b08cfc0a1e3cfffed6f79ae88311db"
server_key_iv: "b055ba380214de0daea54adf82cf72ff"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.467757 16671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:00.494913 16671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:00.495679 16671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:00.495888 16671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:00.503299 16671 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:45219
I20260805 23:06:00.503330 16772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:45219 every 8 connection(s)
I20260805 23:06:00.504282 16671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:06:00.506670 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 16671
I20260805 23:06:00.506804 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:06:00.507119 3121 external_mini_cluster.cc:1428] Setting key c89aa6d6203416d5d4fcddb0c2a93bf1
I20260805 23:06:00.507324 16773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:00.513978 16773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23: Bootstrap starting.
I20260805 23:06:00.516481 16773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:00.516489 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:00.508473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.3.12.126:45219)
0805 23:06:00.509100 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:00.509116 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:00.509152 (+ 36us) server_negotiation.cc:407] Connection header received
0805 23:06:00.509833 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:00.509850 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:00.510242 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:00.510587 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:00.511004 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.511999 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.512957 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.513437 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.513627 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:00.514564 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:00.514569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:00.514582 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:06:00.514993 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:00.515055 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:06:00.515078 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:00.515207 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:00.515354 (+ 147us) server_negotiation.cc:299] Negotiation successful
0805 23:06:00.515652 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":145,"threads_started":1}
I20260805 23:06:00.517320 16773 log.cc:824] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:00.519639 16773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23: No bootstrap required, opened a new log
I20260805 23:06:00.523456 16773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dddcc49085e43cab42be18f9085bd23" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45219 } }
I20260805 23:06:00.523756 16773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:00.523844 16773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dddcc49085e43cab42be18f9085bd23, State: Initialized, Role: FOLLOWER
I20260805 23:06:00.524377 16773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dddcc49085e43cab42be18f9085bd23" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45219 } }
I20260805 23:06:00.524541 16773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:06:00.524631 16773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:06:00.524735 16773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:00.525720 16773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dddcc49085e43cab42be18f9085bd23" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45219 } }
I20260805 23:06:00.526075 16773 leader_election.cc:302] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dddcc49085e43cab42be18f9085bd23; no voters:
I20260805 23:06:00.526414 16773 leader_election.cc:288] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:06:00.526614 16778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:06:00.526866 16778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [term 1 LEADER]: Becoming Leader. State: Replica: 7dddcc49085e43cab42be18f9085bd23, State: Running, Role: LEADER
I20260805 23:06:00.527220 16778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dddcc49085e43cab42be18f9085bd23" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45219 } }
I20260805 23:06:00.527798 16773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:06:00.529320 16781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dddcc49085e43cab42be18f9085bd23. Latest consensus state: current_term: 1 leader_uuid: "7dddcc49085e43cab42be18f9085bd23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dddcc49085e43cab42be18f9085bd23" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45219 } } }
I20260805 23:06:00.529366 16779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dddcc49085e43cab42be18f9085bd23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dddcc49085e43cab42be18f9085bd23" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 45219 } } }
I20260805 23:06:00.529456 16781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [sys.catalog]: This master's current role is: LEADER
I20260805 23:06:00.529469 16779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23 [sys.catalog]: This master's current role is: LEADER
I20260805 23:06:00.529911 16796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:06:00.532850 16796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:06:00.538148 16796 catalog_manager.cc:1347] Generated new cluster ID: 7059a4a7ebfb4924a965c07e5b4c8539
I20260805 23:06:00.538218 16796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:06:00.550452 16796 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:06:00.551703 16796 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:06:00.560158 16796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dddcc49085e43cab42be18f9085bd23: Generated new TSK 0
I20260805 23:06:00.561054 16796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:06:00.565802 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45219
--builtin_ntp_servers=127.3.12.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260805 23:06:00.680135 16808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:00.680511 16808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:00.680595 16808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:00.684664 16808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:00.684746 16808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:00.684839 16808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:06:00.689635 16808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:00.690779 16808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:00.692219 16808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:00.702244 16819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:00.702244 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:00.702317 16808 server_base.cc:1034] running on GCE node
W20260805 23:06:00.702267 16816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:00.702914 16808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:00.703547 16808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:00.704727 16808 hybrid_clock.cc:648] HybridClock initialized: now 1785971160704708 us; error 32 us; skew 500 ppm
I20260805 23:06:00.706593 16808 webserver.cc:460] Webserver started at http://127.3.12.65:41291/ using document root <none> and password file <none>
I20260805 23:06:00.707234 16808 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:00.707386 16808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:00.707634 16808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:00.709764 16808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "e382f1dffab34b68985e544b0f471ee5"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "2bace65503007424c4791df582386ba1"
server_key_iv: "6c83faa8b6031840f1cfa5363445e26a"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.710364 16808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "e382f1dffab34b68985e544b0f471ee5"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "2bace65503007424c4791df582386ba1"
server_key_iv: "6c83faa8b6031840f1cfa5363445e26a"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.714350 16808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:06:00.717038 16830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:00.718238 16808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:06:00.718371 16808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "e382f1dffab34b68985e544b0f471ee5"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "2bace65503007424c4791df582386ba1"
server_key_iv: "6c83faa8b6031840f1cfa5363445e26a"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.718489 16808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:00.739215 16808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:00.739948 16808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:00.740159 16808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:00.740798 16808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:00.741771 16808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:00.741842 16808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:00.741907 16808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:00.741948 16808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:00.751845 16808 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:36917
I20260805 23:06:00.751876 16998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:36917 every 8 connection(s)
I20260805 23:06:00.752815 16808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:06:00.753548 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 16808
I20260805 23:06:00.753654 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:06:00.754040 3121 external_mini_cluster.cc:1428] Setting key 0186cc7f292a5e0eee5337dfa812418b
I20260805 23:06:00.756105 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45219
--builtin_ntp_servers=127.3.12.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260805 23:06:00.763000 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:00.754821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:34669 (local address 127.3.12.126:45219)
0805 23:06:00.754976 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:00.754981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:00.756845 (+ 1864us) server_negotiation.cc:407] Connection header received
0805 23:06:00.758250 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:00.758253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:00.758313 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:00.758393 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:00.759258 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.759806 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.761016 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.761249 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.761365 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:00.762480 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:00.762484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:00.762487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:06:00.762501 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:00.762524 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:06:00.762538 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:00.762626 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:00.762790 (+ 164us) server_negotiation.cc:299] Negotiation successful
0805 23:06:00.762854 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 23:06:00.763825 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:00.755331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:45219 (local address 127.3.12.65:34669)
0805 23:06:00.756694 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:00.756734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:00.758037 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:00.758559 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:00.758566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:00.758648 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:00.759121 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:00.759133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.759939 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.759947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:00.760882 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:00.760892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.761405 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.761409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:00.761576 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:00.762218 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:06:00.762220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:00.762259 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:06:00.762322 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:00.762628 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:00.762641 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:00.762650 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 23:06:00.762784 (+ 134us) client_negotiation.cc:240] Negotiation successful
0805 23:06:00.763070 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":132,"threads_started":1}
I20260805 23:06:00.765146 17002 heartbeater.cc:347] Connected to a master server at 127.3.12.126:45219
I20260805 23:06:00.765399 17002 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:00.765931 17002 heartbeater.cc:511] Master 127.3.12.126:45219 requested a full tablet report, sending...
I20260805 23:06:00.767609 16722 ts_manager.cc:194] Registered new tserver with Master: e382f1dffab34b68985e544b0f471ee5 (127.3.12.65:36917)
I20260805 23:06:00.769253 16722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.65:34669
W20260805 23:06:00.869964 17009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:00.870221 17009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:00.870258 17009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:00.874409 17009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:00.874470 17009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:00.874531 17009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:06:00.879158 17009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:00.880192 17009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:00.881561 17009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:00.891317 17020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:00.891345 17018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:00.891328 17016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:00.891400 17009 server_base.cc:1034] running on GCE node
I20260805 23:06:00.892048 17009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:00.892793 17009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:00.894003 17009 hybrid_clock.cc:648] HybridClock initialized: now 1785971160893986 us; error 30 us; skew 500 ppm
I20260805 23:06:00.895931 17009 webserver.cc:460] Webserver started at http://127.3.12.66:40729/ using document root <none> and password file <none>
I20260805 23:06:00.896620 17009 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:00.896719 17009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:00.896966 17009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:00.899339 17009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "663c53ad69254a1cbd6318b0bb72fd91"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "bbc3bc0fa92715415999ddeacd6f7ed1"
server_key_iv: "477d661d9cefb5ed04230f3115029237"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.899940 17009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "663c53ad69254a1cbd6318b0bb72fd91"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "bbc3bc0fa92715415999ddeacd6f7ed1"
server_key_iv: "477d661d9cefb5ed04230f3115029237"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.903764 17009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260805 23:06:00.906667 17032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:00.908217 17009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:06:00.908376 17009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "663c53ad69254a1cbd6318b0bb72fd91"
format_stamp: "Formatted at 2026-08-05 23:06:00 on dist-test-slave-2d19"
server_key: "bbc3bc0fa92715415999ddeacd6f7ed1"
server_key_iv: "477d661d9cefb5ed04230f3115029237"
server_key_version: "encryptionkey@0"
I20260805 23:06:00.908491 17009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:00.918311 17009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:00.919008 17009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:00.919209 17009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:00.919803 17009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:00.920830 17009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:00.920878 17009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:00.920948 17009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:00.920995 17009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:00.930512 17009 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:35061
I20260805 23:06:00.930539 17185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:35061 every 8 connection(s)
I20260805 23:06:00.931483 17009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:06:00.933125 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 17009
I20260805 23:06:00.933259 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:06:00.933631 3121 external_mini_cluster.cc:1428] Setting key 91e99625830d3f6b73b3f7c0e74554fb
I20260805 23:06:00.935926 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45219
--builtin_ntp_servers=127.3.12.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260805 23:06:00.943696 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:00.933619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:37525 (local address 127.3.12.126:45219)
0805 23:06:00.933803 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:00.933808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:00.935253 (+ 1445us) server_negotiation.cc:407] Connection header received
0805 23:06:00.937022 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:00.937025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:00.937087 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:00.937191 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:00.938465 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.939361 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.941297 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.941604 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.941709 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:00.943175 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:00.943178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:00.943180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:06:00.943195 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:00.943214 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:06:00.943222 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:00.943282 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:00.943501 (+ 219us) server_negotiation.cc:299] Negotiation successful
0805 23:06:00.943558 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260805 23:06:00.945142 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:00.934109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:45219 (local address 127.3.12.66:37525)
0805 23:06:00.934826 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:00.934868 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:00.936755 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:00.937451 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:00.937464 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:00.937599 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:00.938303 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:00.938319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.939529 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.939535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:00.941141 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:00.941157 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:00.941818 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:00.941827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:00.942084 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:00.942821 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:06:00.942825 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:00.942892 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:06:00.942998 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:00.943318 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:00.943330 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:00.943339 (+ 9us) client_negotiation.cc:769] Sending connection context
0805 23:06:00.943516 (+ 177us) client_negotiation.cc:240] Negotiation successful
0805 23:06:00.943884 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":118,"threads_started":1}
I20260805 23:06:00.946596 17186 heartbeater.cc:347] Connected to a master server at 127.3.12.126:45219
I20260805 23:06:00.946895 17186 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:00.947538 17186 heartbeater.cc:511] Master 127.3.12.126:45219 requested a full tablet report, sending...
I20260805 23:06:00.948823 16722 ts_manager.cc:194] Registered new tserver with Master: 663c53ad69254a1cbd6318b0bb72fd91 (127.3.12.66:35061)
I20260805 23:06:00.949465 16722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.66:37525
W20260805 23:06:01.064924 17193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:01.065225 17193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:01.065289 17193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:01.069422 17193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:01.069509 17193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:01.069603 17193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:06:01.074635 17193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:01.075786 17193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:01.077346 17193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:06:01.087730 17193 server_base.cc:1034] running on GCE node
W20260805 23:06:01.087688 17202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:01.087688 17203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:01.087899 17206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:01.088526 17193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:01.089196 17193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:01.090400 17193 hybrid_clock.cc:648] HybridClock initialized: now 1785971161090380 us; error 41 us; skew 500 ppm
I20260805 23:06:01.092423 17193 webserver.cc:460] Webserver started at http://127.3.12.67:38599/ using document root <none> and password file <none>
I20260805 23:06:01.093055 17193 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:01.093151 17193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:01.093395 17193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:01.095665 17193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "8b048043522744618f208d94f9c72121"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "43b932fafd7d434666868756bf220634"
server_key_iv: "a534050f22fa893998fb7717551818c3"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.096338 17193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "8b048043522744618f208d94f9c72121"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "43b932fafd7d434666868756bf220634"
server_key_iv: "a534050f22fa893998fb7717551818c3"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.100436 17193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260805 23:06:01.103186 17216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.104629 17193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:06:01.104784 17193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "8b048043522744618f208d94f9c72121"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "43b932fafd7d434666868756bf220634"
server_key_iv: "a534050f22fa893998fb7717551818c3"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.104911 17193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:01.119773 17193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:01.120576 17193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:01.120776 17193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:01.121407 17193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:01.122468 17193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:01.122541 17193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.122615 17193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:01.122668 17193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.132247 17193 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:32839
I20260805 23:06:01.132308 17389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:32839 every 8 connection(s)
I20260805 23:06:01.133214 17193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:06:01.134848 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 17193
I20260805 23:06:01.134974 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:06:01.135295 3121 external_mini_cluster.cc:1428] Setting key 699318d0d757696c4cacad7c95082c1e
I20260805 23:06:01.142911 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:01.135151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:44371 (local address 127.3.12.126:45219)
0805 23:06:01.135321 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:01.135325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:01.136649 (+ 1324us) server_negotiation.cc:407] Connection header received
0805 23:06:01.138140 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:01.138142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:01.138189 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:01.138268 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:01.139181 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:01.139746 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:01.140963 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:01.141220 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:01.141311 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:01.142464 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:01.142467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:01.142469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:06:01.142481 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:01.142499 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:06:01.142519 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:01.142579 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:01.142736 (+ 157us) server_negotiation.cc:299] Negotiation successful
0805 23:06:01.142789 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 23:06:01.143790 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:01.135698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:45219 (local address 127.3.12.67:44371)
0805 23:06:01.136482 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:01.136536 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:01.137914 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:01.138435 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:01.138442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:01.138527 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:01.139037 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:01.139049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:01.139861 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:01.139864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:01.140844 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:01.140855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:01.141361 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:01.141363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:01.141505 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:01.142204 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:06:01.142206 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:01.142248 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:06:01.142316 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:01.142595 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:01.142603 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:01.142607 (+ 4us) client_negotiation.cc:769] Sending connection context
0805 23:06:01.142719 (+ 112us) client_negotiation.cc:240] Negotiation successful
0805 23:06:01.142947 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":149,"threads_started":1}
I20260805 23:06:01.144960 17390 heartbeater.cc:347] Connected to a master server at 127.3.12.126:45219
I20260805 23:06:01.145220 17390 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:01.145773 17390 heartbeater.cc:511] Master 127.3.12.126:45219 requested a full tablet report, sending...
I20260805 23:06:01.146838 16722 ts_manager.cc:194] Registered new tserver with Master: 8b048043522744618f208d94f9c72121 (127.3.12.67:32839)
I20260805 23:06:01.147447 16722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.67:44371
I20260805 23:06:01.149730 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260805 23:06:01.152765 16776 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:06:01.151823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.3.12.126:45219)
0805 23:06:01.151990 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:01.151995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:01.152131 (+ 136us) server_negotiation.cc:407] Connection header received
0805 23:06:01.152228 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:01.152230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:01.152297 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:01.152376 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:01.152632 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57818: BlockingRecv error: recv got EOF from 127.0.0.1:57818 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260805 23:06:01.152871 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 16808
I20260805 23:06:01.164036 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 17009
I20260805 23:06:01.182461 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 17193
I20260805 23:06:01.202800 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 16671
2026-08-05T23:06:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (930 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-05T23:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:06:01Z Disabled control of system clock
I20260805 23:06:01.243405 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42199
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:42199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260805 23:06:01.360575 17414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:01.360849 17414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:01.360888 17414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:01.364948 17414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:06:01.365015 17414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:01.365031 17414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:01.365047 17414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:06:01.365063 17414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:06:01.369874 17414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:42199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:01.371017 17414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:01.372521 17414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:01.381699 17429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:01.381699 17427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:01.381719 17426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:01.381947 17414 server_base.cc:1034] running on GCE node
I20260805 23:06:01.382609 17414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:01.383782 17414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:01.384991 17414 hybrid_clock.cc:648] HybridClock initialized: now 1785971161384973 us; error 37 us; skew 500 ppm
I20260805 23:06:01.387035 17414 webserver.cc:460] Webserver started at http://127.3.12.126:37519/ using document root <none> and password file <none>
I20260805 23:06:01.387661 17414 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:01.387759 17414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:01.388013 17414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:01.390318 17414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "a1fda557e4d04883a692b944a096a6fa"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "c2cdaf484911bdf4e264dbd8bcd5130c"
server_key_iv: "2e277e218ad5538fa899ece55ff9c146"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.390920 17414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "a1fda557e4d04883a692b944a096a6fa"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "c2cdaf484911bdf4e264dbd8bcd5130c"
server_key_iv: "2e277e218ad5538fa899ece55ff9c146"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.394971 17414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:06:01.397573 17438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.398947 17414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:06:01.399106 17414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "a1fda557e4d04883a692b944a096a6fa"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "c2cdaf484911bdf4e264dbd8bcd5130c"
server_key_iv: "2e277e218ad5538fa899ece55ff9c146"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.399231 17414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:01.404074 17414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:01.404804 17414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:01.404999 17414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:01.412096 17414 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:42199
I20260805 23:06:01.412122 17511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:42199 every 8 connection(s)
I20260805 23:06:01.413110 17414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:06:01.416415 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:01.419360 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 17414
I20260805 23:06:01.419510 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:06:01.419792 3121 external_mini_cluster.cc:1428] Setting key e8e78562633b97dec84ef1f296ff3926
I20260805 23:06:01.422498 17512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa: Bootstrap starting.
I20260805 23:06:01.425335 17512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:01.426039 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:01.421333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.3.12.126:42199)
0805 23:06:01.422031 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:01.422052 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:01.422097 (+ 45us) server_negotiation.cc:407] Connection header received
0805 23:06:01.422950 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:01.422973 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:01.423407 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:01.423805 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:01.424529 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:01.424534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:01.424548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:06:01.424561 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:01.424628 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:06:01.424651 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:01.424760 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:01.424878 (+ 118us) server_negotiation.cc:299] Negotiation successful
0805 23:06:01.425122 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":114,"threads_started":1}
I20260805 23:06:01.426265 17512 log.cc:824] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:01.428354 17512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa: No bootstrap required, opened a new log
I20260805 23:06:01.431818 17512 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fda557e4d04883a692b944a096a6fa" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42199 } }
I20260805 23:06:01.432116 17512 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:01.432704 17512 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fda557e4d04883a692b944a096a6fa, State: Initialized, Role: FOLLOWER
I20260805 23:06:01.433281 17512 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fda557e4d04883a692b944a096a6fa" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42199 } }
I20260805 23:06:01.433442 17512 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:06:01.433533 17512 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:06:01.433632 17512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:01.434579 17512 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fda557e4d04883a692b944a096a6fa" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42199 } }
I20260805 23:06:01.434917 17512 leader_election.cc:302] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1fda557e4d04883a692b944a096a6fa; no voters:
I20260805 23:06:01.435242 17512 leader_election.cc:288] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:06:01.435379 17521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:06:01.435621 17521 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [term 1 LEADER]: Becoming Leader. State: Replica: a1fda557e4d04883a692b944a096a6fa, State: Running, Role: LEADER
I20260805 23:06:01.435913 17521 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fda557e4d04883a692b944a096a6fa" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42199 } }
I20260805 23:06:01.436570 17512 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:06:01.437629 17525 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fda557e4d04883a692b944a096a6fa. Latest consensus state: current_term: 1 leader_uuid: "a1fda557e4d04883a692b944a096a6fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fda557e4d04883a692b944a096a6fa" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42199 } } }
I20260805 23:06:01.437758 17525 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [sys.catalog]: This master's current role is: LEADER
I20260805 23:06:01.438097 17523 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1fda557e4d04883a692b944a096a6fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1fda557e4d04883a692b944a096a6fa" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42199 } } }
I20260805 23:06:01.438154 17536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:06:01.438191 17523 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa [sys.catalog]: This master's current role is: LEADER
I20260805 23:06:01.441329 17536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:06:01.446951 17536 catalog_manager.cc:1347] Generated new cluster ID: fdb2ab92aa4542a6a59a7eae0191eaa6
I20260805 23:06:01.447031 17536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:06:01.457353 17536 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:06:01.457683 17536 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:06:01.464206 17536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1fda557e4d04883a692b944a096a6fa: Generated new TSK 0
I20260805 23:06:01.464931 17536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:06:01.469691 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42199
--builtin_ntp_servers=127.3.12.84:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260805 23:06:01.587316 17550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:01.587592 17550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:01.587635 17550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:01.591715 17550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:01.591778 17550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:01.591835 17550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:06:01.596891 17550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:01.598066 17550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:01.599535 17550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:01.609889 17560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:01.609890 17550 server_base.cc:1034] running on GCE node
W20260805 23:06:01.609870 17558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:01.609870 17562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:01.610587 17550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:01.611318 17550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:01.612556 17550 hybrid_clock.cc:648] HybridClock initialized: now 1785971161612542 us; error 28 us; skew 500 ppm
I20260805 23:06:01.614576 17550 webserver.cc:460] Webserver started at http://127.3.12.65:37061/ using document root <none> and password file <none>
I20260805 23:06:01.615188 17550 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:01.615286 17550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:01.615540 17550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:01.617699 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "551017e5939c48ec8f773a35d441d9b3"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "9f46f600b12c396c823545983958b65a"
server_key_iv: "a230a3d0cc507c75134c31fef906e2f6"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.618284 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "551017e5939c48ec8f773a35d441d9b3"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "9f46f600b12c396c823545983958b65a"
server_key_iv: "a230a3d0cc507c75134c31fef906e2f6"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.622009 17550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260805 23:06:01.624600 17570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.625751 17550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:06:01.625890 17550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "551017e5939c48ec8f773a35d441d9b3"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "9f46f600b12c396c823545983958b65a"
server_key_iv: "a230a3d0cc507c75134c31fef906e2f6"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.626008 17550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:01.630609 17550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:01.631208 17550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:01.631392 17550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:01.632004 17550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:01.632970 17550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:01.633040 17550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.633111 17550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:01.633162 17550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.642325 17550 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:42103
I20260805 23:06:01.642390 17750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:42103 every 8 connection(s)
I20260805 23:06:01.643306 17550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:06:01.647149 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 17550
I20260805 23:06:01.647284 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:06:01.647603 3121 external_mini_cluster.cc:1428] Setting key b56cdc2a9b061346a81f6fb213729c70
I20260805 23:06:01.649859 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:01.645197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:46633 (local address 127.3.12.126:42199)
0805 23:06:01.645371 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:01.645376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:01.646449 (+ 1073us) server_negotiation.cc:407] Connection header received
0805 23:06:01.647901 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:01.647905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:01.647952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:01.648065 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:01.649355 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:01.649358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:01.649360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:06:01.649373 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:01.649398 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:06:01.649408 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:01.649501 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:01.649648 (+ 147us) server_negotiation.cc:299] Negotiation successful
0805 23:06:01.649691 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 23:06:01.650012 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42199
--builtin_ntp_servers=127.3.12.84:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260805 23:06:01.650719 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:01.645636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42199 (local address 127.3.12.65:46633)
0805 23:06:01.646295 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:01.646336 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:01.647637 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:01.648294 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:01.648311 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:01.648405 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:01.649017 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:06:01.649021 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:01.649095 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:06:01.649188 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:01.649504 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:01.649512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:01.649516 (+ 4us) client_negotiation.cc:769] Sending connection context
0805 23:06:01.649653 (+ 137us) client_negotiation.cc:240] Negotiation successful
0805 23:06:01.649845 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260805 23:06:01.651947 17752 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42199
I20260805 23:06:01.652174 17752 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:01.652628 17752 heartbeater.cc:511] Master 127.3.12.126:42199 requested a full tablet report, sending...
I20260805 23:06:01.654142 17451 ts_manager.cc:194] Registered new tserver with Master: 551017e5939c48ec8f773a35d441d9b3 (127.3.12.65:42103)
W20260805 23:06:01.764467 17757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:01.764773 17757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:01.764846 17757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:01.769002 17757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:01.769090 17757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:01.769183 17757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:06:01.773900 17757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:01.775079 17757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:01.776549 17757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:06:01.786659 17757 server_base.cc:1034] running on GCE node
W20260805 23:06:01.786619 17764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:01.786607 17765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:01.786607 17769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:01.787308 17757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:01.787985 17757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:01.789175 17757 hybrid_clock.cc:648] HybridClock initialized: now 1785971161789137 us; error 49 us; skew 500 ppm
I20260805 23:06:01.791054 17757 webserver.cc:460] Webserver started at http://127.3.12.66:44565/ using document root <none> and password file <none>
I20260805 23:06:01.791654 17757 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:01.791718 17757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:01.791981 17757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:01.794135 17757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "96a1c647616f421d8b3756da2808b1e7"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "bc73ae87a6edd7d608bb05939edaf841"
server_key_iv: "1045a55aa9eda13aaeeaba45892227fe"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.794736 17757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "96a1c647616f421d8b3756da2808b1e7"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "bc73ae87a6edd7d608bb05939edaf841"
server_key_iv: "1045a55aa9eda13aaeeaba45892227fe"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.798722 17757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260805 23:06:01.801448 17776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.802709 17757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:06:01.802842 17757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "96a1c647616f421d8b3756da2808b1e7"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "bc73ae87a6edd7d608bb05939edaf841"
server_key_iv: "1045a55aa9eda13aaeeaba45892227fe"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.802961 17757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:01.807641 17757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:01.808295 17757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:01.808486 17757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:01.809042 17757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:01.810055 17757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:01.810127 17757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.810199 17757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:01.810242 17757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.819126 17757 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:42327
I20260805 23:06:01.819151 17925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:42327 every 8 connection(s)
I20260805 23:06:01.820058 17757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:06:01.826745 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:01.821947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:54803 (local address 127.3.12.126:42199)
0805 23:06:01.822125 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:01.822129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:01.823189 (+ 1060us) server_negotiation.cc:407] Connection header received
0805 23:06:01.825078 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:01.825081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:01.825121 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:01.825199 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:01.826352 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:01.826355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:01.826357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:06:01.826366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:01.826382 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:06:01.826389 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:01.826462 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:01.826593 (+ 131us) server_negotiation.cc:299] Negotiation successful
0805 23:06:01.826631 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260805 23:06:01.827684 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 17757
I20260805 23:06:01.827698 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:01.822333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42199 (local address 127.3.12.66:54803)
0805 23:06:01.823039 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:01.823081 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:01.824772 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:01.825367 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:01.825374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:01.825462 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:01.826100 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:06:01.826102 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:01.826143 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:06:01.826213 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:01.826467 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:01.826475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:01.826479 (+ 4us) client_negotiation.cc:769] Sending connection context
0805 23:06:01.826612 (+ 133us) client_negotiation.cc:240] Negotiation successful
0805 23:06:01.826840 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260805 23:06:01.827781 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:06:01.828148 3121 external_mini_cluster.cc:1428] Setting key 965984ad8cc7fdfc22912fb9b4f0d26b
I20260805 23:06:01.829030 17926 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42199
I20260805 23:06:01.829284 17926 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:01.829701 17926 heartbeater.cc:511] Master 127.3.12.126:42199 requested a full tablet report, sending...
I20260805 23:06:01.830303 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42199
--builtin_ntp_servers=127.3.12.84:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260805 23:06:01.830897 17451 ts_manager.cc:194] Registered new tserver with Master: 96a1c647616f421d8b3756da2808b1e7 (127.3.12.66:42327)
W20260805 23:06:01.946949 17931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:01.947247 17931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:01.947316 17931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:01.951395 17931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:01.951495 17931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:01.951583 17931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:06:01.956451 17931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:01.957577 17931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:01.959024 17931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:01.969069 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:01.969169 17931 server_base.cc:1034] running on GCE node
W20260805 23:06:01.969251 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:01.969383 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:01.969779 17931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:01.970468 17931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:01.971665 17931 hybrid_clock.cc:648] HybridClock initialized: now 1785971161971636 us; error 50 us; skew 500 ppm
I20260805 23:06:01.973603 17931 webserver.cc:460] Webserver started at http://127.3.12.67:43491/ using document root <none> and password file <none>
I20260805 23:06:01.974232 17931 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:01.974335 17931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:01.974576 17931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:01.976708 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "c9f7f98f1d7e41178feb00e90f0caa85"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "3a5d424a5302e69123fff859b80958d0"
server_key_iv: "427ac2552a02e75e9e45aa7b179f341e"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.977277 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "c9f7f98f1d7e41178feb00e90f0caa85"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "3a5d424a5302e69123fff859b80958d0"
server_key_iv: "427ac2552a02e75e9e45aa7b179f341e"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.981443 17931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:06:01.984062 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.985327 17931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:06:01.985467 17931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "c9f7f98f1d7e41178feb00e90f0caa85"
format_stamp: "Formatted at 2026-08-05 23:06:01 on dist-test-slave-2d19"
server_key: "3a5d424a5302e69123fff859b80958d0"
server_key_iv: "427ac2552a02e75e9e45aa7b179f341e"
server_key_version: "encryptionkey@0"
I20260805 23:06:01.985592 17931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:01.990247 17931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:01.990863 17931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:01.991062 17931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:01.991632 17931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:01.992650 17931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:01.992734 17931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:01.992823 17931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:01.992874 17931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:02.001884 17931 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:32781
I20260805 23:06:02.001971 18111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:32781 every 8 connection(s)
I20260805 23:06:02.002971 17931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:06:02.007813 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 17931
I20260805 23:06:02.007937 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:06:02.008337 3121 external_mini_cluster.cc:1428] Setting key 107768607928ccbb09d5d273922372fa
I20260805 23:06:02.010226 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:02.005024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:53497 (local address 127.3.12.126:42199)
0805 23:06:02.005172 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:02.005176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:02.006413 (+ 1237us) server_negotiation.cc:407] Connection header received
0805 23:06:02.008110 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:02.008113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:02.008154 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:02.008235 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:02.009708 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:02.009711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:02.009714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:06:02.009726 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:02.009749 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:06:02.009759 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:02.009859 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:02.010038 (+ 179us) server_negotiation.cc:299] Negotiation successful
0805 23:06:02.010075 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:06:02.011209 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:02.005455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42199 (local address 127.3.12.67:53497)
0805 23:06:02.006256 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:02.006299 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:02.007781 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:02.008468 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:02.008478 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:02.008596 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:02.009387 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:06:02.009389 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:02.009430 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:06:02.009527 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:02.009870 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:02.009880 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:02.009886 (+ 6us) client_negotiation.cc:769] Sending connection context
0805 23:06:02.010090 (+ 204us) client_negotiation.cc:240] Negotiation successful
0805 23:06:02.010345 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":106,"threads_started":1}
I20260805 23:06:02.012362 18112 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42199
I20260805 23:06:02.012610 18112 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:02.013042 18112 heartbeater.cc:511] Master 127.3.12.126:42199 requested a full tablet report, sending...
I20260805 23:06:02.014051 17451 ts_manager.cc:194] Registered new tserver with Master: c9f7f98f1d7e41178feb00e90f0caa85 (127.3.12.67:32781)
I20260805 23:06:02.022559 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260805 23:06:02.025542 17519 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:06:02.024594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.3.12.126:42199)
0805 23:06:02.024768 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:02.024771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:02.024865 (+ 94us) server_negotiation.cc:407] Connection header received
0805 23:06:02.024970 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:02.024972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:02.025022 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:02.025091 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:02.025409 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50582: BlockingRecv error: recv got EOF from 127.0.0.1:50582 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260805 23:06:02.025956 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 17550
I20260805 23:06:02.046372 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 17757
I20260805 23:06:02.058965 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 17931
I20260805 23:06:02.074268 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 17414
2026-08-05T23:06:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (870 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:06:02 dist-test-slave-2d19 krb5kdc[18123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:06:02 dist-test-slave-2d19 krb5kdc[18123](info): set up 2 sockets
Aug 05 23:06:02 dist-test-slave-2d19 krb5kdc[18123](info): commencing operation
krb5kdc: starting...
W20260805 23:06:03.312619 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:06:03 dist-test-slave-2d19 krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:06:03.447517 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:34649
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:34649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:06:03.563853 18142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:03.564133 18142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:03.564177 18142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:03.568325 18142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:06:03.568396 18142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:03.568415 18142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:03.568455 18142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:06:03.568488 18142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:06:03.574020 18142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:34649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:34649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:03.575234 18142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:03.576403 18142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:03.585403 18155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:03.585367 18152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:03.585639 18142 server_base.cc:1034] running on GCE node
W20260805 23:06:03.585435 18153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:03.586390 18142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:03.587610 18142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:03.588811 18142 hybrid_clock.cc:648] HybridClock initialized: now 1785971163588782 us; error 46 us; skew 500 ppm
Aug 05 23:06:03 dist-test-slave-2d19 krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:03.591833 18142 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:06:03.592906 18142 webserver.cc:460] Webserver started at http://127.3.12.126:33959/ using document root <none> and password file <none>
I20260805 23:06:03.593505 18142 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:03.593604 18142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:03.593851 18142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:03.596536 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "179851d719b0459c911e15c7faa888ed"
format_stamp: "Formatted at 2026-08-05 23:06:03 on dist-test-slave-2d19"
server_key: "cdb681ef4dbf75a876ed64819a43105c"
server_key_iv: "adb9611079018979fe31b609628c4e59"
server_key_version: "encryptionkey@0"
I20260805 23:06:03.597134 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "179851d719b0459c911e15c7faa888ed"
format_stamp: "Formatted at 2026-08-05 23:06:03 on dist-test-slave-2d19"
server_key: "cdb681ef4dbf75a876ed64819a43105c"
server_key_iv: "adb9611079018979fe31b609628c4e59"
server_key_version: "encryptionkey@0"
I20260805 23:06:03.601107 18142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:06:03.603626 18162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:03.604971 18142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:06:03.605116 18142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "179851d719b0459c911e15c7faa888ed"
format_stamp: "Formatted at 2026-08-05 23:06:03 on dist-test-slave-2d19"
server_key: "cdb681ef4dbf75a876ed64819a43105c"
server_key_iv: "adb9611079018979fe31b609628c4e59"
server_key_version: "encryptionkey@0"
I20260805 23:06:03.605234 18142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:03.624372 18142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:03.627861 18142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:03.628093 18142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:03.635429 18142 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:34649
I20260805 23:06:03.635730 18229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:34649 every 8 connection(s)
I20260805 23:06:03.636539 18142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:06:03.639364 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:03.645040 18230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed: Bootstrap starting.
I20260805 23:06:03.645295 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 18142
I20260805 23:06:03.645435 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:06:03.645711 3121 external_mini_cluster.cc:1428] Setting key e79cabc567955f825cc74eabb0693a76
I20260805 23:06:03.647823 18230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:03.648661 18230 log.cc:824] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:03.650837 18230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed: No bootstrap required, opened a new log
Aug 05 23:06:03 dist-test-slave-2d19 krb5kdc[18123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:06:03.654614 18230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179851d719b0459c911e15c7faa888ed" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34649 } }
I20260805 23:06:03.654907 18230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:03.654984 18230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179851d719b0459c911e15c7faa888ed, State: Initialized, Role: FOLLOWER
I20260805 23:06:03.655444 18230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179851d719b0459c911e15c7faa888ed" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34649 } }
I20260805 23:06:03.655604 18230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:06:03.655702 18230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:06:03.655823 18230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:03.656874 18230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179851d719b0459c911e15c7faa888ed" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34649 } }
I20260805 23:06:03.657235 18230 leader_election.cc:302] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179851d719b0459c911e15c7faa888ed; no voters:
I20260805 23:06:03.657618 18230 leader_election.cc:288] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:06:03.657691 18237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:06:03.657939 18237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [term 1 LEADER]: Becoming Leader. State: Replica: 179851d719b0459c911e15c7faa888ed, State: Running, Role: LEADER
I20260805 23:06:03.658319 18237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179851d719b0459c911e15c7faa888ed" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34649 } }
I20260805 23:06:03.659076 18230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:06:03.660317 18239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179851d719b0459c911e15c7faa888ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179851d719b0459c911e15c7faa888ed" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34649 } } }
I20260805 23:06:03.660322 18238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179851d719b0459c911e15c7faa888ed. Latest consensus state: current_term: 1 leader_uuid: "179851d719b0459c911e15c7faa888ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179851d719b0459c911e15c7faa888ed" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 34649 } } }
I20260805 23:06:03.660567 18239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [sys.catalog]: This master's current role is: LEADER
I20260805 23:06:03.660573 18238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed [sys.catalog]: This master's current role is: LEADER
I20260805 23:06:03.661034 18246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:06:03.661443 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:03.647238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.3.12.126:34649)
0805 23:06:03.647678 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:03.647695 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:03.647730 (+ 35us) server_negotiation.cc:407] Connection header received
0805 23:06:03.648432 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:03.648450 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:03.648791 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:03.649098 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:03.649936 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:03.650943 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:03.651832 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:03.652412 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:03.652605 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:03.655513 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:03.655518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:03.655533 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:03.655562 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:03.657662 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:03.658287 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:03.658294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:03.658301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:03.658478 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:03.658784 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:03.658788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:03.658790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:03.659310 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:03.659518 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:03.660020 (+ 502us) server_negotiation.cc:299] Negotiation successful
0805 23:06:03.660394 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":146,"threads_started":1}
I20260805 23:06:03.664573 18246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:06:03.670296 18246 catalog_manager.cc:1347] Generated new cluster ID: 2597aefe404a4a9d95b5b8d5123ef4e6
I20260805 23:06:03.670378 18246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:06:03.692451 18246 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:06:03.693508 18246 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:06:03.706089 18246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed: Generated new TSK 0
I20260805 23:06:03.706948 18246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:06:03.743237 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34649
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:06:03.860985 18258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:03.861243 18258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:03.861290 18258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:03.865711 18258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:03.865777 18258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:03.865840 18258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:06:03.871454 18258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:03.872558 18258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:03.873548 18258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:06:03.883420 18258 server_base.cc:1034] running on GCE node
W20260805 23:06:03.883355 18265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:03.883351 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:03.883351 18266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:03.883970 18258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:03.884748 18258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:03.885965 18258 hybrid_clock.cc:648] HybridClock initialized: now 1785971163885943 us; error 35 us; skew 500 ppm
Aug 05 23:06:03 dist-test-slave-2d19 krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:03.888935 18258 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:06:03.889853 18258 webserver.cc:460] Webserver started at http://127.3.12.65:35821/ using document root <none> and password file <none>
I20260805 23:06:03.890376 18258 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:03.890447 18258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:03.890614 18258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:03.892714 18258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "5fa6ecdd02db4795a1196ac5788feece"
format_stamp: "Formatted at 2026-08-05 23:06:03 on dist-test-slave-2d19"
server_key: "118248b0aadcacf87bcfee65a71f6db8"
server_key_iv: "011930046b32c7455354ee4408754f94"
server_key_version: "encryptionkey@0"
I20260805 23:06:03.893244 18258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "5fa6ecdd02db4795a1196ac5788feece"
format_stamp: "Formatted at 2026-08-05 23:06:03 on dist-test-slave-2d19"
server_key: "118248b0aadcacf87bcfee65a71f6db8"
server_key_iv: "011930046b32c7455354ee4408754f94"
server_key_version: "encryptionkey@0"
I20260805 23:06:03.897044 18258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 23:06:03.899454 18278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:03.900815 18258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:06:03.900939 18258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "5fa6ecdd02db4795a1196ac5788feece"
format_stamp: "Formatted at 2026-08-05 23:06:03 on dist-test-slave-2d19"
server_key: "118248b0aadcacf87bcfee65a71f6db8"
server_key_iv: "011930046b32c7455354ee4408754f94"
server_key_version: "encryptionkey@0"
I20260805 23:06:03.901051 18258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:03.912050 18258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:03.915423 18258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:03.915637 18258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:03.916211 18258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:03.917079 18258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:03.917128 18258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:03.917169 18258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:03.917186 18258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:03.925554 18258 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:33735
I20260805 23:06:03.925654 18429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:33735 every 8 connection(s)
I20260805 23:06:03.926476 18258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:06:03.930506 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 18258
I20260805 23:06:03.930642 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:06:03.930991 3121 external_mini_cluster.cc:1428] Setting key 3ba8629a80f686d251e5c44f8d354792
Aug 05 23:06:03 dist-test-slave-2d19 krb5kdc[18123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:06:03.942651 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:03.928418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:57769 (local address 127.3.12.126:34649)
0805 23:06:03.928582 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:03.928585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:03.929346 (+ 761us) server_negotiation.cc:407] Connection header received
0805 23:06:03.930917 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:03.930920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:03.931016 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:03.931130 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:03.932699 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:03.933246 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:03.935037 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:03.935271 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:03.935385 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:03.938294 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:03.938302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:03.938306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:03.938340 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:03.940072 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:03.940788 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:03.940792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:03.940794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:03.940915 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:03.941317 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:03.941322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:03.941324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:03.941603 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:03.941716 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:03.942283 (+ 567us) server_negotiation.cc:299] Negotiation successful
0805 23:06:03.942441 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:06:03.943470 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:03.928736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34649 (local address 127.3.12.65:57769)
0805 23:06:03.929185 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:03.929230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:03.930685 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:03.931369 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:03.931380 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:03.931888 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:03.932543 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:03.932566 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:03.933383 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:03.933389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:03.934905 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:03.934916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:03.935431 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:03.935435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:03.935612 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:03.936293 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:06:03.936319 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:03.938111 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:03.940235 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:03.940241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:03.940326 (+ 85us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:03.940652 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:03.941028 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:03.941031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:03.941032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:03.941194 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:03.941741 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:03.941747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:03.941991 (+ 244us) client_negotiation.cc:769] Sending connection context
0805 23:06:03.942159 (+ 168us) client_negotiation.cc:240] Negotiation successful
0805 23:06:03.942426 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
I20260805 23:06:03.945130 18430 heartbeater.cc:347] Connected to a master server at 127.3.12.126:34649
I20260805 23:06:03.945417 18430 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:03.945982 18430 heartbeater.cc:511] Master 127.3.12.126:34649 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:06:03.947692 18187 ts_manager.cc:194] Registered new tserver with Master: 5fa6ecdd02db4795a1196ac5788feece (127.3.12.65:33735)
I20260805 23:06:03.949363 18187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:57769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:06:03.963290 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34649
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:06:04.076385 18437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:04.076647 18437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:04.076689 18437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:04.080687 18437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:04.080740 18437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:04.080798 18437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:06:04.085812 18437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:04.086784 18437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:04.087754 18437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:04.097725 18443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:04.097741 18444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:04.097728 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:04.098109 18437 server_base.cc:1034] running on GCE node
I20260805 23:06:04.098666 18437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:04.099319 18437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:04.100565 18437 hybrid_clock.cc:648] HybridClock initialized: now 1785971164100548 us; error 36 us; skew 500 ppm
Aug 05 23:06:04 dist-test-slave-2d19 krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:04.109251 18437 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:06:04.110376 18437 webserver.cc:460] Webserver started at http://127.3.12.66:42595/ using document root <none> and password file <none>
I20260805 23:06:04.110999 18437 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:04.111099 18437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:04.111352 18437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:04.113608 18437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "2603649f962c4a208f974d34d2178a2f"
format_stamp: "Formatted at 2026-08-05 23:06:04 on dist-test-slave-2d19"
server_key: "cc18797228783b18dad11c8b17110480"
server_key_iv: "326850aa98e44da90187364faa2dc8d6"
server_key_version: "encryptionkey@0"
I20260805 23:06:04.114216 18437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "2603649f962c4a208f974d34d2178a2f"
format_stamp: "Formatted at 2026-08-05 23:06:04 on dist-test-slave-2d19"
server_key: "cc18797228783b18dad11c8b17110480"
server_key_iv: "326850aa98e44da90187364faa2dc8d6"
server_key_version: "encryptionkey@0"
I20260805 23:06:04.118245 18437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:06:04.120832 18457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:04.122066 18437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:06:04.122210 18437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "2603649f962c4a208f974d34d2178a2f"
format_stamp: "Formatted at 2026-08-05 23:06:04 on dist-test-slave-2d19"
server_key: "cc18797228783b18dad11c8b17110480"
server_key_iv: "326850aa98e44da90187364faa2dc8d6"
server_key_version: "encryptionkey@0"
I20260805 23:06:04.122324 18437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:04.139508 18437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:04.142591 18437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:04.142809 18437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:04.143427 18437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:04.144315 18437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:04.144389 18437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:04.144474 18437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:04.144518 18437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:04.154029 18437 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:37249
I20260805 23:06:04.154081 18618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:37249 every 8 connection(s)
I20260805 23:06:04.154999 18437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:06:04.160755 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 18437
I20260805 23:06:04.160887 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:06:04.161260 3121 external_mini_cluster.cc:1428] Setting key e632535802521132f0fb36a13d3b2eaa
Aug 05 23:06:04 dist-test-slave-2d19 krb5kdc[18123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:06:04.171485 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.157134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:44887 (local address 127.3.12.126:34649)
0805 23:06:04.157295 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.157298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.158178 (+ 880us) server_negotiation.cc:407] Connection header received
0805 23:06:04.159726 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.159730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.159795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.159926 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:04.161581 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.162251 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.163897 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.164124 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.164212 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.167403 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:04.167410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:04.167416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:04.167451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:04.169307 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:04.169878 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:04.169882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:04.169884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:04.169971 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:04.170334 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:04.170339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:04.170341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:04.170538 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:04.170649 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.171176 (+ 527us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.171317 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:06:04.172451 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.157497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34649 (local address 127.3.12.66:44887)
0805 23:06:04.157999 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:04.158057 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:04.159500 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:04.160122 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:04.160129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:04.160695 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:04.161347 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.161375 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.162399 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.162402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.163754 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.163769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.164312 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.164319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.164552 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.165355 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:06:04.165385 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:04.167220 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:04.169455 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:04.169468 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:04.169482 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:04.169766 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:04.170076 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:04.170080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:04.170082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:04.170233 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:04.170650 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:04.170658 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:04.170932 (+ 274us) client_negotiation.cc:769] Sending connection context
0805 23:06:04.171135 (+ 203us) client_negotiation.cc:240] Negotiation successful
0805 23:06:04.171484 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":129,"threads_started":1}
I20260805 23:06:04.173911 18619 heartbeater.cc:347] Connected to a master server at 127.3.12.126:34649
I20260805 23:06:04.174209 18619 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:04.174750 18619 heartbeater.cc:511] Master 127.3.12.126:34649 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
I20260805 23:06:04.175922 18187 ts_manager.cc:194] Registered new tserver with Master: 2603649f962c4a208f974d34d2178a2f (127.3.12.66:37249)
I20260805 23:06:04.176585 18187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:44887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:06:04.195016 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34649
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:06:04.311568 18625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:04.311836 18625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:04.311879 18625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:04.315897 18625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:04.315955 18625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:04.316015 18625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:06:04.321256 18625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:34649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:04.322312 18625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:04.323343 18625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:04.333031 18637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:04.333033 18634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:04.333098 18635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:04.333238 18625 server_base.cc:1034] running on GCE node
I20260805 23:06:04.333741 18625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:04.334359 18625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:04.335564 18625 hybrid_clock.cc:648] HybridClock initialized: now 1785971164335556 us; error 34 us; skew 500 ppm
Aug 05 23:06:04 dist-test-slave-2d19 krb5kdc[18123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:04.338399 18625 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:06:04.339272 18625 webserver.cc:460] Webserver started at http://127.3.12.67:45737/ using document root <none> and password file <none>
I20260805 23:06:04.339799 18625 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:04.339859 18625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:04.340023 18625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:06:04.342136 18625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "56f8d6391fa14a728b5285fd324b3f50"
format_stamp: "Formatted at 2026-08-05 23:06:04 on dist-test-slave-2d19"
server_key: "d9b5c3dd94cd1e30d4bada9c3933d34b"
server_key_iv: "88c85381d613702629d4e12593efb6e1"
server_key_version: "encryptionkey@0"
I20260805 23:06:04.342675 18625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "56f8d6391fa14a728b5285fd324b3f50"
format_stamp: "Formatted at 2026-08-05 23:06:04 on dist-test-slave-2d19"
server_key: "d9b5c3dd94cd1e30d4bada9c3933d34b"
server_key_iv: "88c85381d613702629d4e12593efb6e1"
server_key_version: "encryptionkey@0"
I20260805 23:06:04.346474 18625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:06:04.349747 18645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:04.351331 18625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:06:04.351444 18625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "56f8d6391fa14a728b5285fd324b3f50"
format_stamp: "Formatted at 2026-08-05 23:06:04 on dist-test-slave-2d19"
server_key: "d9b5c3dd94cd1e30d4bada9c3933d34b"
server_key_iv: "88c85381d613702629d4e12593efb6e1"
server_key_version: "encryptionkey@0"
I20260805 23:06:04.351540 18625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:04.363808 18625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:04.366803 18625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:04.366967 18625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:04.367511 18625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:04.368395 18625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:04.368444 18625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:04.368486 18625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:04.368502 18625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:04.377408 18625 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:41463
I20260805 23:06:04.377504 18793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:41463 every 8 connection(s)
I20260805 23:06:04.378386 18625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:06:04.382603 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 18625
I20260805 23:06:04.382728 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:06:04.383033 3121 external_mini_cluster.cc:1428] Setting key f39fe9f7bee7341afe90f0b61319f961
Aug 05 23:06:04 dist-test-slave-2d19 krb5kdc[18123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:06:04.392488 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.380340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:46239 (local address 127.3.12.126:34649)
0805 23:06:04.380534 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.380539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.381382 (+ 843us) server_negotiation.cc:407] Connection header received
0805 23:06:04.382865 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.382869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.382930 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.383041 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:04.384547 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.385086 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.386268 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.386489 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.386628 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.389160 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:04.389163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:04.389165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:04.389190 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:04.390528 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:04.391022 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:04.391025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:04.391026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:04.391101 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:04.391410 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:04.391412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:04.391413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:04.391595 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:04.391669 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.392162 (+ 493us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.392323 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260805 23:06:04.393394 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.380764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:34649 (local address 127.3.12.67:46239)
0805 23:06:04.381231 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:04.381273 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:04.382599 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:04.383232 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:04.383239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:04.383718 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:04.384382 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.384406 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.385220 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.385223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.386128 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.386138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.386672 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.386676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.386857 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.387419 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:06:04.387435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:04.388993 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:04.390652 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:04.390657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:04.390671 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:04.390926 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:04.391178 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:04.391180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:04.391182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:04.391324 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:04.391681 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:04.391686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:04.391934 (+ 248us) client_negotiation.cc:769] Sending connection context
0805 23:06:04.392090 (+ 156us) client_negotiation.cc:240] Negotiation successful
0805 23:06:04.392480 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260805 23:06:04.394587 18795 heartbeater.cc:347] Connected to a master server at 127.3.12.126:34649
I20260805 23:06:04.394810 18795 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:04.395304 18795 heartbeater.cc:511] Master 127.3.12.126:34649 requested a full tablet report, sending...
I20260805 23:06:04.396473 18187 ts_manager.cc:194] Registered new tserver with Master: 56f8d6391fa14a728b5285fd324b3f50 (127.3.12.67:41463)
I20260805 23:06:04.397153 18187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:46239
I20260805 23:06:04.397167 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:06:04.407407 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.399502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.3.12.126:34649)
0805 23:06:04.399660 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.399664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.399825 (+ 161us) server_negotiation.cc:407] Connection header received
0805 23:06:04.399981 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.399984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.400067 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.400154 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:04.400897 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.401405 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.402266 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.402493 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.402596 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.404018 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:04.404022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:04.404024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:04.404049 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:04.405519 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:04.406084 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:04.406090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:04.406092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:04.406181 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:04.406492 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:04.406496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:04.406498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:04.406724 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:04.406870 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.407114 (+ 244us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.407235 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:06:04.410488 18187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:06:04.413664 18187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:06:04.428311 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.422487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.3.12.67:41463)
0805 23:06:04.423087 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.423103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.423130 (+ 27us) server_negotiation.cc:407] Connection header received
0805 23:06:04.423214 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.423219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.423794 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.423994 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:04.424321 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.425252 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.426800 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.427500 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.427648 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.427809 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.428017 (+ 208us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.428130 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":177,"threads_started":1}
I20260805 23:06:04.428341 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.422337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:41463 (local address 127.0.0.1:36452)
0805 23:06:04.422659 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:04.422705 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:04.422824 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:04.423981 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:04.423987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:04.424012 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:04.424166 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.424172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.425453 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.425459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.426650 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.426665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.427674 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.427678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.427786 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.427828 (+ 42us) client_negotiation.cc:769] Sending connection context
0805 23:06:04.427968 (+ 140us) client_negotiation.cc:240] Negotiation successful
0805 23:06:04.428092 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260805 23:06:04.428743 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.423385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:33735 (local address 127.0.0.1:48370)
0805 23:06:04.423684 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:04.423698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:04.423791 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:04.424518 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:04.424521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:04.424539 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:04.424621 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.424627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.426040 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.426044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.427198 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.427206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.428328 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.428331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.428450 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.428487 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 23:06:04.428538 (+ 51us) client_negotiation.cc:240] Negotiation successful
0805 23:06:04.428600 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":23,"thread_start_us":86,"threads_started":1}
I20260805 23:06:04.429344 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.423531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.3.12.65:33735)
0805 23:06:04.424041 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.424056 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.424079 (+ 23us) server_negotiation.cc:407] Connection header received
0805 23:06:04.424157 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.424162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.424377 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.424529 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:04.424752 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.425883 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.427320 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.428145 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.428909 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.429079 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.429128 (+ 49us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.429212 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":102,"threads_started":1}
I20260805 23:06:04.429417 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.423038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:37249 (local address 127.0.0.1:51692)
0805 23:06:04.423616 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:04.423633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:04.423726 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:04.424170 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:04.424172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:04.424189 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:04.424315 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.424322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.425761 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.425764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.427085 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.427096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.428400 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.428403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.428512 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.428544 (+ 32us) client_negotiation.cc:769] Sending connection context
0805 23:06:04.429192 (+ 648us) client_negotiation.cc:240] Negotiation successful
0805 23:06:04.429272 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260805 23:06:04.429852 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.423238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.3.12.66:37249)
0805 23:06:04.423697 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.423713 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.423740 (+ 27us) server_negotiation.cc:407] Connection header received
0805 23:06:04.423821 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.423828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.424018 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.424178 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:04.424410 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.425609 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.427231 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.428069 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.428171 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.428328 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.429642 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.429746 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":111,"threads_started":1}
I20260805 23:06:04.432137 18707 tablet_service.cc:1467] Processing CreateTablet for tablet 85cc6425b9004274bfcf0a0144a3fc51 (DEFAULT_TABLE table=test-table [id=f13184b0e12948bdab7101f1de9473bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:06:04.432282 18348 tablet_service.cc:1467] Processing CreateTablet for tablet 85cc6425b9004274bfcf0a0144a3fc51 (DEFAULT_TABLE table=test-table [id=f13184b0e12948bdab7101f1de9473bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:06:04.432631 18526 tablet_service.cc:1467] Processing CreateTablet for tablet 85cc6425b9004274bfcf0a0144a3fc51 (DEFAULT_TABLE table=test-table [id=f13184b0e12948bdab7101f1de9473bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:06:04.433393 18707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cc6425b9004274bfcf0a0144a3fc51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:04.433391 18348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cc6425b9004274bfcf0a0144a3fc51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:04.433535 18526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cc6425b9004274bfcf0a0144a3fc51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:04.439626 18816 tablet_bootstrap.cc:492] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Bootstrap starting.
I20260805 23:06:04.441250 18817 tablet_bootstrap.cc:492] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece: Bootstrap starting.
I20260805 23:06:04.441251 18818 tablet_bootstrap.cc:492] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Bootstrap starting.
I20260805 23:06:04.442373 18816 tablet_bootstrap.cc:654] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:04.443353 18816 log.cc:824] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:04.443624 18817 tablet_bootstrap.cc:654] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:04.444120 18818 tablet_bootstrap.cc:654] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:04.445009 18818 log.cc:824] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:04.445010 18817 log.cc:824] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:04.445546 18816 tablet_bootstrap.cc:492] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: No bootstrap required, opened a new log
I20260805 23:06:04.445760 18816 ts_tablet_manager.cc:1397] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:06:04.446738 18817 tablet_bootstrap.cc:492] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece: No bootstrap required, opened a new log
I20260805 23:06:04.446955 18817 ts_tablet_manager.cc:1397] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:06:04.447023 18818 tablet_bootstrap.cc:492] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: No bootstrap required, opened a new log
I20260805 23:06:04.447201 18818 ts_tablet_manager.cc:1397] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:06:04.449783 18817 raft_consensus.cc:348] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.449772 18816 raft_consensus.cc:348] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.450028 18816 raft_consensus.cc:374] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:04.450035 18817 raft_consensus.cc:374] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:04.450093 18817 raft_consensus.cc:729] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa6ecdd02db4795a1196ac5788feece, State: Initialized, Role: FOLLOWER
I20260805 23:06:04.450093 18816 raft_consensus.cc:729] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f8d6391fa14a728b5285fd324b3f50, State: Initialized, Role: FOLLOWER
I20260805 23:06:04.450553 18817 consensus_queue.cc:260] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.450552 18816 consensus_queue.cc:260] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.451447 18795 heartbeater.cc:503] Master 127.3.12.126:34649 was elected leader, sending a full tablet report...
I20260805 23:06:04.451577 18816 ts_tablet_manager.cc:1428] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:06:04.451581 18817 ts_tablet_manager.cc:1428] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260805 23:06:04.451949 18430 heartbeater.cc:503] Master 127.3.12.126:34649 was elected leader, sending a full tablet report...
I20260805 23:06:04.451756 18818 raft_consensus.cc:348] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.452041 18818 raft_consensus.cc:374] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:04.452107 18818 raft_consensus.cc:729] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2603649f962c4a208f974d34d2178a2f, State: Initialized, Role: FOLLOWER
I20260805 23:06:04.452617 18818 consensus_queue.cc:260] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.453253 18619 heartbeater.cc:503] Master 127.3.12.126:34649 was elected leader, sending a full tablet report...
I20260805 23:06:04.453579 18818 ts_tablet_manager.cc:1428] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260805 23:06:04.629452 18797 tablet.cc:2369] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 23:06:04.656278 18621 tablet.cc:2369] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:06:04.661271 18825 raft_consensus.cc:482] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:06:04.661521 18825 raft_consensus.cc:504] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.662768 18825 leader_election.cc:288] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56f8d6391fa14a728b5285fd324b3f50 (127.3.12.67:41463), 5fa6ecdd02db4795a1196ac5788feece (127.3.12.65:33735)
I20260805 23:06:04.666462 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.663140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:45087 (local address 127.3.12.67:41463)
0805 23:06:04.663297 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.663302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.663352 (+ 50us) server_negotiation.cc:407] Connection header received
0805 23:06:04.663418 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.663420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.663475 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.663560 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:04.663882 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.664467 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.665498 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.665898 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.666025 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.666090 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.666272 (+ 182us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.666339 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 23:06:04.666462 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.663013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:41463 (local address 127.3.12.66:45087)
0805 23:06:04.663143 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:04.663161 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:04.663265 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:04.663631 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:04.663634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:04.663659 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:04.663767 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.663772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.664576 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.664581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.665365 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.665374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.666020 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.666023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.666134 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.666164 (+ 30us) client_negotiation.cc:769] Sending connection context
0805 23:06:04.666254 (+ 90us) client_negotiation.cc:240] Negotiation successful
0805 23:06:04.666330 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":2}
I20260805 23:06:04.666903 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.663492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:45399 (local address 127.3.12.65:33735)
0805 23:06:04.663671 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.663676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.663751 (+ 75us) server_negotiation.cc:407] Connection header received
0805 23:06:04.663898 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.663901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.663953 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.664043 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:04.664354 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.664865 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.665892 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.666302 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.666424 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.666481 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.666712 (+ 231us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.666784 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260805 23:06:04.667157 18734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cc6425b9004274bfcf0a0144a3fc51" candidate_uuid: "2603649f962c4a208f974d34d2178a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f8d6391fa14a728b5285fd324b3f50" is_pre_election: true
I20260805 23:06:04.667279 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.663312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:33735 (local address 127.3.12.66:45399)
0805 23:06:04.663646 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:04.663661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:04.663778 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:04.664069 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:04.664072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:04.664088 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:04.664216 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.664221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.664986 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.664992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.665762 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:04.665770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.666430 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.666433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:04.666548 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.666582 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 23:06:04.667064 (+ 482us) client_negotiation.cc:240] Negotiation successful
0805 23:06:04.667140 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":133,"threads_started":1}
I20260805 23:06:04.667532 18734 raft_consensus.cc:2454] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2603649f962c4a208f974d34d2178a2f in term 0.
I20260805 23:06:04.667946 18371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cc6425b9004274bfcf0a0144a3fc51" candidate_uuid: "2603649f962c4a208f974d34d2178a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa6ecdd02db4795a1196ac5788feece" is_pre_election: true
I20260805 23:06:04.668045 18464 leader_election.cc:302] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2603649f962c4a208f974d34d2178a2f, 56f8d6391fa14a728b5285fd324b3f50; no voters:
I20260805 23:06:04.668309 18371 raft_consensus.cc:2454] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2603649f962c4a208f974d34d2178a2f in term 0.
I20260805 23:06:04.668315 18825 raft_consensus.cc:2781] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:06:04.668401 18825 raft_consensus.cc:482] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:06:04.668469 18825 raft_consensus.cc:3037] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:04.669634 18825 raft_consensus.cc:504] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.670027 18825 leader_election.cc:288] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [CANDIDATE]: Term 1 election: Requested vote from peers 56f8d6391fa14a728b5285fd324b3f50 (127.3.12.67:41463), 5fa6ecdd02db4795a1196ac5788feece (127.3.12.65:33735)
I20260805 23:06:04.670429 18371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cc6425b9004274bfcf0a0144a3fc51" candidate_uuid: "2603649f962c4a208f974d34d2178a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa6ecdd02db4795a1196ac5788feece"
I20260805 23:06:04.670437 18734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85cc6425b9004274bfcf0a0144a3fc51" candidate_uuid: "2603649f962c4a208f974d34d2178a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f8d6391fa14a728b5285fd324b3f50"
I20260805 23:06:04.670562 18371 raft_consensus.cc:3037] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:04.670583 18734 raft_consensus.cc:3037] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:04.671667 18734 raft_consensus.cc:2454] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2603649f962c4a208f974d34d2178a2f in term 1.
I20260805 23:06:04.671703 18371 raft_consensus.cc:2454] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2603649f962c4a208f974d34d2178a2f in term 1.
I20260805 23:06:04.672010 18464 leader_election.cc:302] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2603649f962c4a208f974d34d2178a2f, 56f8d6391fa14a728b5285fd324b3f50; no voters:
I20260805 23:06:04.672178 18825 raft_consensus.cc:2781] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:06:04.672456 18825 raft_consensus.cc:686] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 1 LEADER]: Becoming Leader. State: Replica: 2603649f962c4a208f974d34d2178a2f, State: Running, Role: LEADER
I20260805 23:06:04.672806 18825 consensus_queue.cc:237] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } }
I20260805 23:06:04.675937 18186 catalog_manager.cc:5495] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2603649f962c4a208f974d34d2178a2f (127.3.12.66). New cstate: current_term: 1 leader_uuid: "2603649f962c4a208f974d34d2178a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2603649f962c4a208f974d34d2178a2f" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 37249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 } health_report { overall_health: UNKNOWN } } }
W20260805 23:06:04.677810 18431 tablet.cc:2369] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:06:04.720408 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.716843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.3.12.66:37249)
0805 23:06:04.716980 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:04.716983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:04.716997 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:06:04.717085 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:04.717087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:04.717144 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:04.717248 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:06:04.717525 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.718020 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.718900 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:04.719119 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:04.719201 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:04.719491 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:06:04.719920 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:06:04.720017 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:04.720156 (+ 139us) server_negotiation.cc:299] Negotiation successful
0805 23:06:04.720209 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260805 23:06:04.726488 18371 raft_consensus.cc:1261] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece [term 1 FOLLOWER]: Refusing update from remote peer 2603649f962c4a208f974d34d2178a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:06:04.726469 18734 raft_consensus.cc:1261] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 1 FOLLOWER]: Refusing update from remote peer 2603649f962c4a208f974d34d2178a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:06:04.727265 18827 consensus_queue.cc:1035] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "56f8d6391fa14a728b5285fd324b3f50" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:06:04.727442 18825 consensus_queue.cc:1035] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa6ecdd02db4795a1196ac5788feece" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:06:04.735656 18831 mvcc.cc:205] Tried to move back new op lower bound from 7315337890712023040 to 7315337890502381568. Current Snapshot: MvccSnapshot[applied={T|T < 7315337890712023040}]
I20260805 23:06:04.744174 18187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:06:04.744410 18187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:06:04.746809 18187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179851d719b0459c911e15c7faa888ed: Sending DeleteTablet for 3 replicas of tablet 85cc6425b9004274bfcf0a0144a3fc51
I20260805 23:06:04.747673 18707 tablet_service.cc:1514] Processing DeleteTablet for tablet 85cc6425b9004274bfcf0a0144a3fc51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:06:04 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:36452
I20260805 23:06:04.747946 18526 tablet_service.cc:1514] Processing DeleteTablet for tablet 85cc6425b9004274bfcf0a0144a3fc51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:06:04 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:51692
I20260805 23:06:04.747931 18348 tablet_service.cc:1514] Processing DeleteTablet for tablet 85cc6425b9004274bfcf0a0144a3fc51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:06:04 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:48370
I20260805 23:06:04.748448 18844 tablet_replica.cc:330] T 85cc6425b9004274bfcf0a0144a3fc51 P 5fa6ecdd02db4795a1196ac5788feece: stopping tablet replica
I20260805 23:06:04.748647 18845 tablet_replica.cc:330] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: stopping tablet replica
I20260805 23:06:04.748709 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 18258
I20260805 23:06:04.749985 18845 raft_consensus.cc:2229] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:06:04.750445 18845 raft_consensus.cc:2258] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:06:04.754916 18845 ts_tablet_manager.cc:1905] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:06:04.759375 18845 ts_tablet_manager.cc:1918] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:06:04.759670 18845 log.cc:1196] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/85cc6425b9004274bfcf0a0144a3fc51
I20260805 23:06:04.760226 18845 ts_tablet_manager.cc:1939] T 85cc6425b9004274bfcf0a0144a3fc51 P 2603649f962c4a208f974d34d2178a2f: Deleting consensus metadata
I20260805 23:06:04.762233 18167 catalog_manager.cc:4841] TS 2603649f962c4a208f974d34d2178a2f (127.3.12.66:37249): tablet 85cc6425b9004274bfcf0a0144a3fc51 (table test-table [id=f13184b0e12948bdab7101f1de9473bd]) successfully deleted
I20260805 23:06:04.763622 18843 tablet_replica.cc:330] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: stopping tablet replica
I20260805 23:06:04.764015 18843 raft_consensus.cc:2229] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:06:04.764397 18843 raft_consensus.cc:2258] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:06:04.766304 18843 ts_tablet_manager.cc:1905] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:06:04.769164 18843 ts_tablet_manager.cc:1918] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:06:04.769272 18843 log.cc:1196] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/85cc6425b9004274bfcf0a0144a3fc51
I20260805 23:06:04.769625 18843 ts_tablet_manager.cc:1939] T 85cc6425b9004274bfcf0a0144a3fc51 P 56f8d6391fa14a728b5285fd324b3f50: Deleting consensus metadata
I20260805 23:06:04.771032 18167 catalog_manager.cc:4841] TS 56f8d6391fa14a728b5285fd324b3f50 (127.3.12.67:41463): tablet 85cc6425b9004274bfcf0a0144a3fc51 (table test-table [id=f13184b0e12948bdab7101f1de9473bd]) successfully deleted
W20260805 23:06:04.772346 18167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:33735 (error 108)
I20260805 23:06:04.772511 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 18437
I20260805 23:06:04.776919 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:04.776392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:33735 (local address 127.0.0.1:48378)
0805 23:06:04.776665 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:04.776795 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260805 23:06:04.777132 18167 catalog_manager.cc:4584] TS 5fa6ecdd02db4795a1196ac5788feece (127.3.12.65:33735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85cc6425b9004274bfcf0a0144a3fc51: Network error: Client connection negotiation failed: client connection to 127.3.12.65:33735: connect: Connection refused (error 111)
I20260805 23:06:04.785780 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 18625
I20260805 23:06:04.802892 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 18142
2026-08-05T23:06:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2730 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:06:04 dist-test-slave-2d19 krb5kdc[18852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:06:04 dist-test-slave-2d19 krb5kdc[18852](info): set up 2 sockets
Aug 05 23:06:04 dist-test-slave-2d19 krb5kdc[18852](info): commencing operation
krb5kdc: starting...
W20260805 23:06:06.043741 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:06:06 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:06:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:06:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/rangeradmin_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/rangeradmin_127.3.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/rangerlookup_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/rangerlookup_127.3.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/HTTP_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/HTTP_127.3.12.84@KRBTEST.COM.keytab.
I20260805 23:06:06.239218 3121 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqJ6txj/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/postgres -l logfile start
2026-08-05 23:06:09.039 UTC [18896] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-05 23:06:09.039 UTC [18896] LOG: listening on IPv4 address "127.3.12.84", port 34719
2026-08-05 23:06:09.043 UTC [18896] LOG: listening on Unix socket "/tmp/.s.PGSQL.34719"
2026-08-05 23:06:09.069 UTC [18902] LOG: database system was shut down at 2026-08-05 23:06:07 UTC
2026-08-05 23:06:09.073 UTC [18896] LOG: database system is ready to accept connections
I20260805 23:06:10.200366 3121 mini_postgres.cc:96] Postgres bound to 34719
2026-08-05 23:06:10.208 UTC [18912] FATAL: database "slave" does not exist
127.3.12.84:34719 - accepting connections
I20260805 23:06:10.209064 3121 mini_ranger.cc:161] Starting Ranger...
I20260805 23:06:10.227201 3121 mini_ranger.cc:84] Created miniranger Postgres user
I20260805 23:06:10.798831 3121 mini_ranger.cc:87] Created ranger Postgres database
I20260805 23:06:10.798949 3121 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-admin
2026-08-05 23:06:11,109 [I] DB FLAVOR :POSTGRES
2026-08-05 23:06:11,109 [I] --------- Verifying Ranger DB connection ---------
2026-08-05 23:06:11,109 [I] Checking connection..
2026-08-05 23:06:11,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-05 23:06:11,394 [I] Checking connection passed.
2026-08-05 23:06:11,394 [I] --------- Verifying version history table ---------
2026-08-05 23:06:11,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 23:06:11,652 [I] Table x_db_version_h does not exist in database ranger
2026-08-05 23:06:11,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 23:06:11,931 [I] Table x_db_version_h does not exist in database ranger
2026-08-05 23:06:11,931 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-05 23:06:11,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-05 23:06:12.173 UTC [18998] WARNING: there is no transaction in progress
2026-08-05 23:06:12,184 [I] create_dbversion_catalog.sql file imported successfully
2026-08-05 23:06:12,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 23:06:12,459 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-05 23:06:12,459 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-05 23:06:12,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-05 23:06:12,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-05 23:06:12,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2d19.c.gcp-upstream.internal','N') ;"
2026-08-05 23:06:13,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-05 23:06:13,515 [I] Table x_portal_user does not exist in database ranger
2026-08-05 23:06:13,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-05 23:06:13,758 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-05 23:06:13,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-05 23:06:14,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-05 23:06:14,280 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-05 23:06:14,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-05 23:06:15.313 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.330 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.349 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.367 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.384 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.406 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.468 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.476 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.499 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.521 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.542 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.571 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.590 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.602 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.613 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:15.626 UTC [19138] WARNING: there is no transaction in progress
2026-08-05 23:06:16,587 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-05 23:06:16,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-05 23:06:16,843 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-05 23:06:16,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-05 23:06:17,142 [I] Table x_portal_user already exists in database 'ranger'
2026-08-05 23:06:17,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-05 23:06:17,413 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-05 23:06:17,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-05 23:06:17,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-05 23:06:17,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2d19.c.gcp-upstream.internal';"
2026-08-05 23:06:18,244 [I] CORE_DB_SCHEMA import status has been updated
2026-08-05 23:06:18,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-05 23:06:18,518 [I] DB_PATCHES have already been applied
I20260805 23:06:18.525187 3121 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqJ6txj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260805 23:06:18.525293 3121 mini_ranger.cc:193] Using host: 127.3.12.84
I20260805 23:06:18.530035 3121 mini_ranger.cc:227] Ranger admin URL: http://127.3.12.84:58929
Aug 05, 2026 11:06:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 05, 2026 11:06:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 05, 2026 11:06:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-admin/ews/webapp
Aug 05, 2026 11:06:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 05, 2026 11:06:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 05, 2026 11:06:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-admin/ews/webapp].
Aug 05, 2026 11:06:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260805 23:06:19.868427 3121 mini_ranger.cc:160] Time spent starting Ranger: real 9.659s user 0.002s sys 0.011s
Aug 05, 2026 11:06:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 05, 2026 11:06:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 05, 2026 11:06:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 05, 2026 11:06:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 05, 2026 11:06:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260805 23:06:42.113965 3121 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/rangerkms_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/rangerkms_127.3.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.12.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.12.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.12.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/HTTP_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.12.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/HTTP_127.3.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 05 23:06:42 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971202, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260805 23:06:42.213773 3121 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260805 23:06:42.226498 3121 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260805 23:06:43.132802 3121 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260805 23:06:43.132911 3121 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms
I20260805 23:06:43.133024 3121 mini_ranger_kms.cc:227] Using postgres at 127.3.12.84:34719
2026-08-05 23:06:43,345 [I] DB FLAVOR :POSTGRES
2026-08-05 23:06:43,345 [I] --------- Verifying Ranger DB connection ---------
2026-08-05 23:06:43,345 [I] Checking connection
2026-08-05 23:06:43,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-05 23:06:43,624 [I] connection success
2026-08-05 23:06:43,625 [I] --------- Verifying Ranger DB tables ---------
2026-08-05 23:06:43,625 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-05 23:06:43,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-05 23:06:43,868 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-05 23:06:43,869 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-05 23:06:43,869 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-05 23:06:43,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:34719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-05 23:06:44,152 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260805 23:06:44.656023 3121 mini_ranger_kms.cc:310] Created kms service
I20260805 23:06:44.851758 3121 mini_ranger_kms.cc:326] Created kudu user
I20260805 23:06:44.899310 3121 mini_ranger_kms.cc:343] Created rangerkms user
Aug 05 23:06:44 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971202, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:06:45.359357 3121 mini_ranger_kms.cc:384] Added ranger policy
I20260805 23:06:45.359517 3121 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260805 23:06:45.359539 3121 mini_ranger_kms.cc:240] Using host: 127.3.12.84
I20260805 23:06:45.364396 3121 mini_ranger_kms.cc:276] Ranger KMS PID: 19378
I20260805 23:06:45.364544 3121 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.12.84:39365
Aug 05, 2026 11:06:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 05, 2026 11:06:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 05, 2026 11:06:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 05, 2026 11:06:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 05, 2026 11:06:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-kms/ews/webapp].
Aug 05, 2026 11:06:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 05 23:06:47 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785971207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.12.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:48.128979 3121 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.915s user 0.013s sys 0.183s
I20260805 23:06:48.129144 3121 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260805 23:06:48.129266 3121 mini_ranger_kms.cc:401] 127.3.12.84:39365/kms/v1/keys
Aug 05 23:06:48 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.12.84@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
Aug 05 23:06:48 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971207, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.12.84@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
Aug 05 23:06:51 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:06:51.068825 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43757
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:43757
--ranger_config_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:39365/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:06:51.182296 19456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:51.182578 19456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:51.182626 19456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:51.186554 19456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:06:51.186617 19456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:51.186637 19456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:51.186653 19456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:06:51.186668 19456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:06:51.192024 19456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.12.84:39365/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:43757
--ranger_config_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:51.193414 19456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:51.194821 19456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:51.206207 19465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:51.206270 19463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:51.206393 19456 server_base.cc:1034] running on GCE node
W20260805 23:06:51.206259 19462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:51.207144 19456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:51.208330 19456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:51.209524 19456 hybrid_clock.cc:648] HybridClock initialized: now 1785971211209496 us; error 42 us; skew 500 ppm
Aug 05 23:06:51 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971211, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:51.212430 19456 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:06:51.213366 19456 webserver.cc:460] Webserver started at http://127.3.12.126:38745/ using document root <none> and password file <none>
I20260805 23:06:51.213968 19456 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:51.214052 19456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:51.214272 19456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 23:06:51 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971211, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:06:51.265928 19456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "dc66969683064f52a2c93909405fa3fb"
format_stamp: "Formatted at 2026-08-05 23:06:51 on dist-test-slave-2d19"
server_key: "bd0e2237985a80036129ef8a4786d0fd"
server_key_iv: "9604b32c730db0b0b89c429c89bcf8e8"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:51.266567 19456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "dc66969683064f52a2c93909405fa3fb"
format_stamp: "Formatted at 2026-08-05 23:06:51 on dist-test-slave-2d19"
server_key: "bd0e2237985a80036129ef8a4786d0fd"
server_key_iv: "9604b32c730db0b0b89c429c89bcf8e8"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:51.270785 19456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:06:51.289353 19473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:51.290789 19456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:06:51.290905 19456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "dc66969683064f52a2c93909405fa3fb"
format_stamp: "Formatted at 2026-08-05 23:06:51 on dist-test-slave-2d19"
server_key: "bd0e2237985a80036129ef8a4786d0fd"
server_key_iv: "9604b32c730db0b0b89c429c89bcf8e8"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:51.291103 19456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:51.325592 19456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:51.328572 19456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:51.328787 19456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:51.336711 19456 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:43757
I20260805 23:06:51.336752 19549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:43757 every 8 connection(s)
I20260805 23:06:51.337682 19456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:06:51.338367 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 19456
I20260805 23:06:51.338508 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:06:51.338861 3121 external_mini_cluster.cc:1428] Setting key 9724081db270aa294b03c5a06dacfad7
I20260805 23:06:51.341143 19551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:51.347046 19551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb: Bootstrap starting.
Aug 05 23:06:51 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:06:51.350124 19551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:51.351519 19551 log.cc:824] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:51.353761 19551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb: No bootstrap required, opened a new log
I20260805 23:06:51.353791 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:51.340378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.3.12.126:43757)
0805 23:06:51.340843 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:51.340858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:51.340895 (+ 37us) server_negotiation.cc:407] Connection header received
0805 23:06:51.341620 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:51.341639 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:51.341971 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:51.342315 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:51.343136 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:51.344198 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:51.345088 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:51.345555 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:51.345730 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:51.348907 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:51.348912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:51.348926 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:51.348954 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:51.350736 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:51.351251 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:51.351256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:51.351261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:51.351364 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:51.351668 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:51.351671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:51.351673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:51.352203 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:51.352438 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:51.352719 (+ 281us) server_negotiation.cc:299] Negotiation successful
0805 23:06:51.353005 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":164,"threads_started":1}
I20260805 23:06:51.356891 19551 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc66969683064f52a2c93909405fa3fb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43757 } }
I20260805 23:06:51.357115 19551 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:51.357187 19551 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc66969683064f52a2c93909405fa3fb, State: Initialized, Role: FOLLOWER
I20260805 23:06:51.357643 19551 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc66969683064f52a2c93909405fa3fb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43757 } }
I20260805 23:06:51.357769 19551 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:06:51.357847 19551 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:06:51.357944 19551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:51.358881 19551 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc66969683064f52a2c93909405fa3fb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43757 } }
I20260805 23:06:51.359223 19551 leader_election.cc:302] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc66969683064f52a2c93909405fa3fb; no voters:
I20260805 23:06:51.359536 19551 leader_election.cc:288] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:06:51.359755 19558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:06:51.360113 19558 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [term 1 LEADER]: Becoming Leader. State: Replica: dc66969683064f52a2c93909405fa3fb, State: Running, Role: LEADER
I20260805 23:06:51.360437 19558 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc66969683064f52a2c93909405fa3fb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43757 } }
I20260805 23:06:51.360857 19551 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:06:51.362039 19559 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc66969683064f52a2c93909405fa3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc66969683064f52a2c93909405fa3fb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43757 } } }
I20260805 23:06:51.362167 19559 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [sys.catalog]: This master's current role is: LEADER
I20260805 23:06:51.362532 19560 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc66969683064f52a2c93909405fa3fb. Latest consensus state: current_term: 1 leader_uuid: "dc66969683064f52a2c93909405fa3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc66969683064f52a2c93909405fa3fb" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43757 } } }
I20260805 23:06:51.362638 19560 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb [sys.catalog]: This master's current role is: LEADER
I20260805 23:06:51.362591 19571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:06:51.363224 19551 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260805 23:06:51.368983 19571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:06:51.376410 19571 catalog_manager.cc:1347] Generated new cluster ID: fb8a99469154404b8bcb4a3df1e84c9b
I20260805 23:06:51.376492 19571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:06:51.395828 19571 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:06:51.397280 19571 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:06:51.409394 19571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb: Generated new TSK 0
I20260805 23:06:51.410283 19571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:06:52.133800 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:39365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43757
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:06:52.307229 19617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:52.307646 19617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:52.307732 19617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:52.315034 19617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:52.315132 19617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:52.315232 19617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:06:52.324177 19617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.12.84:39365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:52.325774 19617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:52.327232 19617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:52.341519 19633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:52.341744 19625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:52.341977 19626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:52.342895 19617 server_base.cc:1034] running on GCE node
I20260805 23:06:52.343415 19617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:52.344326 19617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:52.348160 19617 hybrid_clock.cc:648] HybridClock initialized: now 1785971212348019 us; error 148 us; skew 500 ppm
Aug 05 23:06:52 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:52.356664 19617 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:06:52.357789 19617 webserver.cc:460] Webserver started at http://127.3.12.65:34481/ using document root <none> and password file <none>
I20260805 23:06:52.358412 19617 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:52.358510 19617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:52.358758 19617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 23:06:52 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:06:52.375916 19617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "dd8be7513dd4411dac39b4b938f1373e"
format_stamp: "Formatted at 2026-08-05 23:06:52 on dist-test-slave-2d19"
server_key: "246b52505b4e96d71983fe1b84ae73d8"
server_key_iv: "f83e76917c2f6fe3ff41c92abee8ded3"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:52.376744 19617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "dd8be7513dd4411dac39b4b938f1373e"
format_stamp: "Formatted at 2026-08-05 23:06:52 on dist-test-slave-2d19"
server_key: "246b52505b4e96d71983fe1b84ae73d8"
server_key_iv: "f83e76917c2f6fe3ff41c92abee8ded3"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:52.386065 19617 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260805 23:06:52.406940 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:52.408804 19617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260805 23:06:52.408951 19617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "dd8be7513dd4411dac39b4b938f1373e"
format_stamp: "Formatted at 2026-08-05 23:06:52 on dist-test-slave-2d19"
server_key: "246b52505b4e96d71983fe1b84ae73d8"
server_key_iv: "f83e76917c2f6fe3ff41c92abee8ded3"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:52.409106 19617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:52.432886 19617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:52.437108 19617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:52.437335 19617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:52.438184 19617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:52.439332 19617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:52.439409 19617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:52.439483 19617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:52.439512 19617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:52.451833 19617 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:42671
I20260805 23:06:52.453122 19617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:06:52.459123 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 19617
I20260805 23:06:52.459261 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:06:52.459589 3121 external_mini_cluster.cc:1428] Setting key 0e41787a7164bcfd33a9d431ae8459f2
I20260805 23:06:52.462488 19823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:42671 every 8 connection(s)
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
Aug 05 23:06:52 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:06:52.497654 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:52.466425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:37391 (local address 127.3.12.126:43757)
0805 23:06:52.468865 (+ 2440us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:52.468870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:52.472364 (+ 3494us) server_negotiation.cc:407] Connection header received
0805 23:06:52.472432 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:52.472435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:52.472501 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:52.472615 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:52.479099 (+ 6484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:52.479988 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:52.481811 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:52.482176 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:52.482266 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:52.492189 (+ 9923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:52.492194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:52.492197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:52.492241 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:52.494403 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:52.495383 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:52.495387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:52.495389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:52.495491 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:52.496077 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:52.496080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:52.496082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:52.496364 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:52.496455 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:52.497309 (+ 854us) server_negotiation.cc:299] Negotiation successful
0805 23:06:52.497457 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2340,"thread_start_us":127,"threads_started":1}
I20260805 23:06:52.499866 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:52.467539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43757 (local address 127.3.12.65:37391)
0805 23:06:52.469186 (+ 1647us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:52.469252 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:52.470492 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:52.477502 (+ 7010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:52.477516 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:52.477951 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:52.478831 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:52.478855 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:52.480197 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:52.480201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:52.481646 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:52.481659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:52.482564 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:52.482568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:52.482779 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:52.483591 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:06:52.483614 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:52.489101 (+ 5487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:52.494632 (+ 5531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:52.494641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:52.494663 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:52.495241 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:52.495665 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:52.495668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:52.495670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:52.495899 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:52.496531 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:52.496539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:52.496967 (+ 428us) client_negotiation.cc:769] Sending connection context
0805 23:06:52.498315 (+ 1348us) client_negotiation.cc:240] Negotiation successful
0805 23:06:52.498746 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"thread_start_us":177,"threads_started":1}
I20260805 23:06:52.503775 19824 heartbeater.cc:347] Connected to a master server at 127.3.12.126:43757
I20260805 23:06:52.504067 19824 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:52.504740 19824 heartbeater.cc:511] Master 127.3.12.126:43757 requested a full tablet report, sending...
I20260805 23:06:52.506664 19496 ts_manager.cc:194] Registered new tserver with Master: dd8be7513dd4411dac39b4b938f1373e (127.3.12.65:42671)
I20260805 23:06:52.508399 19496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:37391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:06:52.520726 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:39365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43757
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 23:06:52 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260805 23:06:52.706717 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:52.707072 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:52.707144 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:52.712981 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:52.713086 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:52.713173 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:06:52.720927 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.12.84:39365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:52.722312 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:52.723624 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:52.736426 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:52.736615 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:52.737568 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:52.737779 19837 server_base.cc:1034] running on GCE node
I20260805 23:06:52.738359 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:52.739136 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:52.740346 19837 hybrid_clock.cc:648] HybridClock initialized: now 1785971212740276 us; error 81 us; skew 500 ppm
Aug 05 23:06:52 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:52.744138 19837 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:06:52.745257 19837 webserver.cc:460] Webserver started at http://127.3.12.66:35779/ using document root <none> and password file <none>
I20260805 23:06:52.745963 19837 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:52.746064 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:52.746301 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 23:06:52 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:06:52.766307 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "c8331e2f7f6b4edd85a068bd5f32b125"
format_stamp: "Formatted at 2026-08-05 23:06:52 on dist-test-slave-2d19"
server_key: "9d3f11a9bde2fe11fb2df724e748e5de"
server_key_iv: "e634019ba43276a3d937a3a444cd1f06"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:52.767079 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "c8331e2f7f6b4edd85a068bd5f32b125"
format_stamp: "Formatted at 2026-08-05 23:06:52 on dist-test-slave-2d19"
server_key: "9d3f11a9bde2fe11fb2df724e748e5de"
server_key_iv: "e634019ba43276a3d937a3a444cd1f06"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:52.777922 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.007s
I20260805 23:06:52.797514 19859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:52.799382 19837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260805 23:06:52.799530 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "c8331e2f7f6b4edd85a068bd5f32b125"
format_stamp: "Formatted at 2026-08-05 23:06:52 on dist-test-slave-2d19"
server_key: "9d3f11a9bde2fe11fb2df724e748e5de"
server_key_iv: "e634019ba43276a3d937a3a444cd1f06"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:52.799687 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:52.834162 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:52.838115 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:52.838325 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:52.839088 19837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:52.840205 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:52.840297 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:52.840365 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:52.840399 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:52.855290 19837 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:42013
I20260805 23:06:52.859138 20014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:42013 every 8 connection(s)
I20260805 23:06:52.859691 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:06:52.861829 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 19837
I20260805 23:06:52.861974 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:06:52.862361 3121 external_mini_cluster.cc:1428] Setting key b7153b8397c8d43bd107dd0ecd62cff4
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
Aug 05 23:06:52 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
I20260805 23:06:52.894935 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:52.864608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:39507 (local address 127.3.12.126:43757)
0805 23:06:52.864916 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:52.864921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:52.872898 (+ 7977us) server_negotiation.cc:407] Connection header received
0805 23:06:52.874374 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:52.874378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:52.874440 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:52.874512 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:52.876514 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:52.877441 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:52.882261 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:52.882642 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:52.882730 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:52.889617 (+ 6887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:52.889622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:52.889625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:52.889664 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:52.891741 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:52.892641 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:52.892645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:52.892647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:52.892743 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:52.893208 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:52.893211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:52.893213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:52.893467 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:52.893724 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:52.894563 (+ 839us) server_negotiation.cc:299] Negotiation successful
0805 23:06:52.894759 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260805 23:06:52.897035 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:52.863170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43757 (local address 127.3.12.66:39507)
0805 23:06:52.872669 (+ 9499us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:52.872758 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:52.874074 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:52.875027 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:52.875038 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:52.875450 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:52.876286 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:52.876311 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:52.877632 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:52.877635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:52.879116 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:52.879131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:52.882964 (+ 3833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:52.882968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:52.883205 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:52.883931 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:06:52.883953 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:52.889314 (+ 5361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:52.891999 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:52.892009 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:52.892049 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:52.892498 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:52.892859 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:52.892862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:52.892864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:52.893084 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:52.893818 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:52.893826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:52.894219 (+ 393us) client_negotiation.cc:769] Sending connection context
0805 23:06:52.895472 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0805 23:06:52.895869 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9178,"thread_start_us":150,"threads_started":1}
I20260805 23:06:52.898715 20015 heartbeater.cc:347] Connected to a master server at 127.3.12.126:43757
I20260805 23:06:52.898970 20015 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:52.899567 20015 heartbeater.cc:511] Master 127.3.12.126:43757 requested a full tablet report, sending...
I20260805 23:06:52.900821 19496 ts_manager.cc:194] Registered new tserver with Master: c8331e2f7f6b4edd85a068bd5f32b125 (127.3.12.66:42013)
I20260805 23:06:52.901592 19496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:39507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:06:52.919533 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:39365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43757
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:06:53.079878 20021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:06:53.080387 20021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:06:53.080516 20021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:06:53.086942 20021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:06:53.087106 20021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:06:53.087247 20021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:06:53.095321 20021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.12.84:39365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:06:53.096967 20021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:06:53.098420 20021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:06:53.111263 20031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:53.111367 20027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:06:53.111236 20029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:06:53.112100 20021 server_base.cc:1034] running on GCE node
I20260805 23:06:53.112592 20021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:06:53.113371 20021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:06:53.114982 20021 hybrid_clock.cc:648] HybridClock initialized: now 1785971213114947 us; error 51 us; skew 500 ppm
Aug 05 23:06:53 dist-test-slave-2d19 krb5kdc[18852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971213, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:06:53.118886 20021 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:06:53.119896 20021 webserver.cc:460] Webserver started at http://127.3.12.67:40687/ using document root <none> and password file <none>
I20260805 23:06:53.120568 20021 fs_manager.cc:359] Metadata directory not provided
I20260805 23:06:53.120657 20021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:06:53.120882 20021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 23:06:53 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971213, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:06:53.136641 20021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "ef142913200d40758ed2f49364e7b7e7"
format_stamp: "Formatted at 2026-08-05 23:06:53 on dist-test-slave-2d19"
server_key: "c56a25608d5f79a5b517c4cb990baf9f"
server_key_iv: "47bd830e93f0d8b997fedf78761751ff"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:53.138604 20021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "ef142913200d40758ed2f49364e7b7e7"
format_stamp: "Formatted at 2026-08-05 23:06:53 on dist-test-slave-2d19"
server_key: "c56a25608d5f79a5b517c4cb990baf9f"
server_key_iv: "47bd830e93f0d8b997fedf78761751ff"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:53.143904 20021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
Aug 05 23:06:53 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:06:53.171492 20040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:53.173256 20021 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260805 23:06:53.173390 20021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "ef142913200d40758ed2f49364e7b7e7"
format_stamp: "Formatted at 2026-08-05 23:06:53 on dist-test-slave-2d19"
server_key: "c56a25608d5f79a5b517c4cb990baf9f"
server_key_iv: "47bd830e93f0d8b997fedf78761751ff"
server_key_version: "kuduclusterkey@0"
I20260805 23:06:53.173537 20021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:06:53.196730 20021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:06:53.200421 20021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:06:53.200642 20021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:06:53.201333 20021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:06:53.202376 20021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:06:53.202450 20021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:53.202502 20021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:06:53.202533 20021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:06:53.214672 20021 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:45273
I20260805 23:06:53.215961 20021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:06:53.219625 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 20021
I20260805 23:06:53.219753 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:06:53.220046 3121 external_mini_cluster.cc:1428] Setting key ef400f4aa775538f9f3deee1b32185b5
I20260805 23:06:53.229382 20201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:45273 every 8 connection(s)
Aug 05 23:06:53 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971213, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:06:53.254166 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.233749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:54421 (local address 127.3.12.126:43757)
0805 23:06:53.233901 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:53.233908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:53.241008 (+ 7100us) server_negotiation.cc:407] Connection header received
0805 23:06:53.241070 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:53.241073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:53.241131 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:53.241216 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:53.243140 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.244015 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.245794 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.246192 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.246269 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.249561 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:53.249566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:53.249569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:53.249599 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:53.251561 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:53.252269 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:53.252273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:53.252275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:53.252366 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:53.252793 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:53.252797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:53.252798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:53.253024 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:53.253105 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:53.253855 (+ 750us) server_negotiation.cc:299] Negotiation successful
0805 23:06:53.253995 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:06:53.255873 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.232419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43757 (local address 127.3.12.67:54421)
0805 23:06:53.238669 (+ 6250us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:53.238740 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:53.239894 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:53.241832 (+ 1938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:53.241842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:53.242240 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:06:53.242948 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.242967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.244188 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.244191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.245644 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.245654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.246498 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.246502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.246719 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.247519 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:06:53.247541 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:06:53.249348 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:06:53.251714 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:53.251720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:53.251740 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:53.252128 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:53.252489 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:06:53.252492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:06:53.252494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:06:53.252682 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:06:53.253174 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:06:53.253181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:06:53.253555 (+ 374us) client_negotiation.cc:769] Sending connection context
0805 23:06:53.254421 (+ 866us) client_negotiation.cc:240] Negotiation successful
0805 23:06:53.254801 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5964,"thread_start_us":214,"threads_started":1}
I20260805 23:06:53.257426 20202 heartbeater.cc:347] Connected to a master server at 127.3.12.126:43757
I20260805 23:06:53.257699 20202 heartbeater.cc:464] Registering TS with master...
I20260805 23:06:53.258285 20202 heartbeater.cc:511] Master 127.3.12.126:43757 requested a full tablet report, sending...
I20260805 23:06:53.259430 19496 ts_manager.cc:194] Registered new tserver with Master: ef142913200d40758ed2f49364e7b7e7 (127.3.12.67:45273)
I20260805 23:06:53.260202 19496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:54421
I20260805 23:06:53.260700 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:06:53.277505 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.263321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.3.12.126:43757)
0805 23:06:53.263436 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:53.263440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:53.264068 (+ 628us) server_negotiation.cc:407] Connection header received
0805 23:06:53.264267 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:53.264271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:53.264318 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:53.264368 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:06:53.265340 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.266145 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.267360 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.267705 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.267775 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.273521 (+ 5746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:06:53.273525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:06:53.273528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:06:53.273558 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:06:53.275504 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:53.276030 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:53.276035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:53.276037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:53.276146 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:06:53.276483 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:06:53.276487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:06:53.276488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:06:53.276710 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:06:53.276814 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:53.277205 (+ 391us) server_negotiation.cc:299] Negotiation successful
0805 23:06:53.277332 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260805 23:06:53.281232 19496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:06:53.285430 19496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:06:53.307096 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.296485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:42013 (local address 127.0.0.1:35098)
0805 23:06:53.299298 (+ 2813us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:53.299312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:53.299414 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:53.300629 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:53.300636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:53.300661 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:53.300809 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.300816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.302527 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.302530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.304639 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.304652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.306346 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.306350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.306505 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.306574 (+ 69us) client_negotiation.cc:769] Sending connection context
0805 23:06:53.306715 (+ 141us) client_negotiation.cc:240] Negotiation successful
0805 23:06:53.306890 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2753,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260805 23:06:53.307653 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.297253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.3.12.66:42013)
0805 23:06:53.299109 (+ 1856us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:53.299127 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:53.299422 (+ 295us) server_negotiation.cc:407] Connection header received
0805 23:06:53.299531 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:53.299537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:53.300142 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:53.300387 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:53.300962 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.302351 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.304863 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.305960 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.306078 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.306268 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:53.307319 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0805 23:06:53.307449 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1746,"thread_start_us":89,"threads_started":1}
I20260805 23:06:53.310851 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.296412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:45273 (local address 127.0.0.1:52398)
0805 23:06:53.298625 (+ 2213us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:53.298684 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:53.298827 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:53.301239 (+ 2412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:53.301242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:53.301259 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:53.301357 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.301362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.303534 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.303537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.304894 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.304907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.310431 (+ 5524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.310435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.310558 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.310592 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 23:06:53.310645 (+ 53us) client_negotiation.cc:240] Negotiation successful
0805 23:06:53.310716 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1988,"mutex_wait_us":43,"thread_start_us":105,"threads_started":1}
I20260805 23:06:53.311501 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.297260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.3.12.67:45273)
0805 23:06:53.299912 (+ 2652us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:53.299924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:53.299948 (+ 24us) server_negotiation.cc:407] Connection header received
0805 23:06:53.300024 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:53.300029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:53.301083 (+ 1054us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:53.301510 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:53.301649 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.302845 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.305097 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.310297 (+ 5200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.311041 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.311191 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:53.311243 (+ 52us) server_negotiation.cc:299] Negotiation successful
0805 23:06:53.311334 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2548,"thread_start_us":142,"threads_started":1}
I20260805 23:06:53.311023 19927 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6cd955908e4e1fad58c8a969131ead (DEFAULT_TABLE table=test-table [id=f5901f7bd2f440079546b92156b2e230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:06:53.314800 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.298365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42671 (local address 127.0.0.1:57756)
0805 23:06:53.299586 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:53.299598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:53.299671 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:53.307721 (+ 8050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:53.307724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:53.307748 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:53.308482 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.308499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.311775 (+ 3276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.311778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.313036 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.313048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.314328 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.314332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.314430 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.314455 (+ 25us) client_negotiation.cc:769] Sending connection context
0805 23:06:53.314518 (+ 63us) client_negotiation.cc:240] Negotiation successful
0805 23:06:53.314585 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":108,"threads_started":1}
I20260805 23:06:53.315616 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.298801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.3.12.65:42671)
0805 23:06:53.305666 (+ 6865us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:53.305673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:53.305699 (+ 26us) server_negotiation.cc:407] Connection header received
0805 23:06:53.305776 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:53.305782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:53.307541 (+ 1759us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:53.308688 (+ 1147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:53.308865 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.310037 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.313211 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.314186 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.315085 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.315316 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:53.315371 (+ 55us) server_negotiation.cc:299] Negotiation successful
0805 23:06:53.315464 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6745,"thread_start_us":133,"threads_started":1}
I20260805 23:06:53.318444 19722 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6cd955908e4e1fad58c8a969131ead (DEFAULT_TABLE table=test-table [id=f5901f7bd2f440079546b92156b2e230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:06:53.316016 20112 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6cd955908e4e1fad58c8a969131ead (DEFAULT_TABLE table=test-table [id=f5901f7bd2f440079546b92156b2e230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:06:53.320197 19722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6cd955908e4e1fad58c8a969131ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:53.320408 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6cd955908e4e1fad58c8a969131ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:53.328691 19927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6cd955908e4e1fad58c8a969131ead. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:06:53.332667 20229 tablet_bootstrap.cc:492] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Bootstrap starting.
I20260805 23:06:53.335546 20229 tablet_bootstrap.cc:654] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:53.337941 20230 tablet_bootstrap.cc:492] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e: Bootstrap starting.
I20260805 23:06:53.340329 20230 tablet_bootstrap.cc:654] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:53.341347 20230 log.cc:824] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:53.341955 20229 log.cc:824] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:53.342741 20231 tablet_bootstrap.cc:492] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Bootstrap starting.
I20260805 23:06:53.346194 20231 tablet_bootstrap.cc:654] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Neither blocks nor log segments found. Creating new log.
I20260805 23:06:53.347572 20231 log.cc:824] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Log is configured to *not* fsync() on all Append() calls
I20260805 23:06:53.349833 20231 tablet_bootstrap.cc:492] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: No bootstrap required, opened a new log
I20260805 23:06:53.350062 20231 ts_tablet_manager.cc:1397] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260805 23:06:53.353190 20230 tablet_bootstrap.cc:492] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e: No bootstrap required, opened a new log
I20260805 23:06:53.353180 20229 tablet_bootstrap.cc:492] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: No bootstrap required, opened a new log
I20260805 23:06:53.353403 20230 ts_tablet_manager.cc:1397] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.004s
I20260805 23:06:53.353403 20229 ts_tablet_manager.cc:1397] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20260805 23:06:53.358325 20230 raft_consensus.cc:348] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.358822 20230 raft_consensus.cc:374] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:53.358948 20230 raft_consensus.cc:729] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8be7513dd4411dac39b4b938f1373e, State: Initialized, Role: FOLLOWER
I20260805 23:06:53.359581 20230 consensus_queue.cc:260] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.359638 20231 raft_consensus.cc:348] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.359899 20231 raft_consensus.cc:374] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:53.359963 20231 raft_consensus.cc:729] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8331e2f7f6b4edd85a068bd5f32b125, State: Initialized, Role: FOLLOWER
I20260805 23:06:53.361265 20230 ts_tablet_manager.cc:1428] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260805 23:06:53.361723 19824 heartbeater.cc:503] Master 127.3.12.126:43757 was elected leader, sending a full tablet report...
I20260805 23:06:53.362315 20231 consensus_queue.cc:260] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.363531 20231 ts_tablet_manager.cc:1428] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260805 23:06:53.363912 20015 heartbeater.cc:503] Master 127.3.12.126:43757 was elected leader, sending a full tablet report...
I20260805 23:06:53.362669 20229 raft_consensus.cc:348] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.365355 20229 raft_consensus.cc:374] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:06:53.365473 20229 raft_consensus.cc:729] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef142913200d40758ed2f49364e7b7e7, State: Initialized, Role: FOLLOWER
I20260805 23:06:53.365995 20229 consensus_queue.cc:260] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.367003 20202 heartbeater.cc:503] Master 127.3.12.126:43757 was elected leader, sending a full tablet report...
I20260805 23:06:53.367142 20229 ts_tablet_manager.cc:1428] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
W20260805 23:06:53.465277 19828 tablet.cc:2369] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 05 23:06:53 dist-test-slave-2d19 krb5kdc[18852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971212, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
W20260805 23:06:53.489114 20203 tablet.cc:2369] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:06:53.597321 20237 raft_consensus.cc:482] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:06:53.597609 20237 raft_consensus.cc:504] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.599131 20237 leader_election.cc:288] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef142913200d40758ed2f49364e7b7e7 (127.3.12.67:45273), c8331e2f7f6b4edd85a068bd5f32b125 (127.3.12.66:42013)
W20260805 23:06:53.611423 20016 tablet.cc:2369] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:06:53.613523 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.600282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:52641 (local address 127.3.12.67:45273)
0805 23:06:53.600406 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:53.600411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:53.604309 (+ 3898us) server_negotiation.cc:407] Connection header received
0805 23:06:53.604543 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:53.604546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:53.604598 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:53.604645 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:53.605512 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.606322 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.610796 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.611397 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.611478 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.611554 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:53.613229 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0805 23:06:53.613341 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260805 23:06:53.614420 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.599347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:45273 (local address 127.3.12.65:52641)
0805 23:06:53.604161 (+ 4814us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:53.604180 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:53.604405 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:53.604916 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:53.604919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:53.604938 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:53.605053 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.605059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.606568 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.606571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.607844 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.607855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.611773 (+ 3918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.611777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.611870 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.611892 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:06:53.614058 (+ 2166us) client_negotiation.cc:240] Negotiation successful
0805 23:06:53.614165 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4725,"thread_start_us":109,"threads_started":1}
I20260805 23:06:53.615006 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.600433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:50225 (local address 127.3.12.66:42013)
0805 23:06:53.600560 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:53.600564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:53.605335 (+ 4771us) server_negotiation.cc:407] Connection header received
0805 23:06:53.608028 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:53.608033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:53.608090 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:53.608138 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:06:53.608566 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.609305 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.612039 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.612574 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.612664 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.612745 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:53.614795 (+ 2050us) server_negotiation.cc:299] Negotiation successful
0805 23:06:53.614888 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260805 23:06:53.616343 20137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6cd955908e4e1fad58c8a969131ead" candidate_uuid: "dd8be7513dd4411dac39b4b938f1373e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef142913200d40758ed2f49364e7b7e7" is_pre_election: true
I20260805 23:06:53.616698 20137 raft_consensus.cc:2454] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8be7513dd4411dac39b4b938f1373e in term 0.
I20260805 23:06:53.617358 19649 leader_election.cc:302] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd8be7513dd4411dac39b4b938f1373e, ef142913200d40758ed2f49364e7b7e7; no voters:
I20260805 23:06:53.617697 20237 raft_consensus.cc:2781] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:06:53.617784 20237 raft_consensus.cc:482] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:06:53.617825 20237 raft_consensus.cc:3037] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:53.619356 20237 raft_consensus.cc:504] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.619733 20237 leader_election.cc:288] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [CANDIDATE]: Term 1 election: Requested vote from peers ef142913200d40758ed2f49364e7b7e7 (127.3.12.67:45273), c8331e2f7f6b4edd85a068bd5f32b125 (127.3.12.66:42013)
I20260805 23:06:53.620463 20137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6cd955908e4e1fad58c8a969131ead" candidate_uuid: "dd8be7513dd4411dac39b4b938f1373e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef142913200d40758ed2f49364e7b7e7"
I20260805 23:06:53.620594 20137 raft_consensus.cc:3037] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:53.621919 20137 raft_consensus.cc:2454] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8be7513dd4411dac39b4b938f1373e in term 1.
I20260805 23:06:53.622356 19649 leader_election.cc:302] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd8be7513dd4411dac39b4b938f1373e, ef142913200d40758ed2f49364e7b7e7; no voters:
I20260805 23:06:53.622602 20237 raft_consensus.cc:2781] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:06:53.622830 20237 raft_consensus.cc:686] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [term 1 LEADER]: Becoming Leader. State: Replica: dd8be7513dd4411dac39b4b938f1373e, State: Running, Role: LEADER
I20260805 23:06:53.623207 20237 consensus_queue.cc:237] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } }
I20260805 23:06:53.626796 19495 catalog_manager.cc:5495] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e reported cstate change: term changed from 0 to 1, leader changed from <none> to dd8be7513dd4411dac39b4b938f1373e (127.3.12.65). New cstate: current_term: 1 leader_uuid: "dd8be7513dd4411dac39b4b938f1373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8be7513dd4411dac39b4b938f1373e" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42671 } health_report { overall_health: HEALTHY } } }
I20260805 23:06:53.615630 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.599755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:42013 (local address 127.3.12.65:50225)
0805 23:06:53.605247 (+ 5492us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:53.605260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:06:53.605401 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:06:53.608299 (+ 2898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:06:53.608302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:06:53.608319 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:06:53.608435 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.608440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.609449 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.609452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.610641 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:06:53.610652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.612861 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.612866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:06:53.613037 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.613069 (+ 32us) client_negotiation.cc:769] Sending connection context
0805 23:06:53.615416 (+ 2347us) client_negotiation.cc:240] Negotiation successful
0805 23:06:53.615494 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5429,"thread_start_us":83,"threads_started":1}
I20260805 23:06:53.629171 19932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6cd955908e4e1fad58c8a969131ead" candidate_uuid: "dd8be7513dd4411dac39b4b938f1373e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" is_pre_election: true
I20260805 23:06:53.629531 19932 raft_consensus.cc:2454] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8be7513dd4411dac39b4b938f1373e in term 0.
I20260805 23:06:53.629907 19950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6cd955908e4e1fad58c8a969131ead" candidate_uuid: "dd8be7513dd4411dac39b4b938f1373e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8331e2f7f6b4edd85a068bd5f32b125"
I20260805 23:06:53.630026 19950 raft_consensus.cc:3037] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:06:53.631445 19950 raft_consensus.cc:2454] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8be7513dd4411dac39b4b938f1373e in term 1.
I20260805 23:06:53.696841 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.689553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.3.12.65:42671)
0805 23:06:53.689678 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0805 23:06:53.689684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:06:53.689700 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:06:53.689804 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:06:53.689807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:06:53.689883 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:06:53.690016 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:06:53.690295 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.691161 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.692504 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:06:53.692852 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:06:53.692949 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:06:53.693388 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:06:53.695933 (+ 2545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:06:53.696075 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0805 23:06:53.696576 (+ 501us) server_negotiation.cc:299] Negotiation successful
0805 23:06:53.696646 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260805 23:06:53.704582 19950 raft_consensus.cc:1261] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 1 FOLLOWER]: Refusing update from remote peer dd8be7513dd4411dac39b4b938f1373e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:06:53.705796 20247 consensus_queue.cc:1035] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8331e2f7f6b4edd85a068bd5f32b125" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 42013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 23:06:53.711409 20137 raft_consensus.cc:1261] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 1 FOLLOWER]: Refusing update from remote peer dd8be7513dd4411dac39b4b938f1373e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:06:53.712121 20247 consensus_queue.cc:1035] T 0c6cd955908e4e1fad58c8a969131ead P dd8be7513dd4411dac39b4b938f1373e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef142913200d40758ed2f49364e7b7e7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:06:53.729411 20250 mvcc.cc:205] Tried to move back new op lower bound from 7315338091324563456 to 7315338091004272640. Current Snapshot: MvccSnapshot[applied={T|T < 7315338091324563456}]
I20260805 23:06:53.729797 20253 mvcc.cc:205] Tried to move back new op lower bound from 7315338091324563456 to 7315338091004272640. Current Snapshot: MvccSnapshot[applied={T|T < 7315338091324563456 or (T in {7315338091324563456})}]
I20260805 23:06:53.732769 20251 mvcc.cc:205] Tried to move back new op lower bound from 7315338091324563456 to 7315338091004272640. Current Snapshot: MvccSnapshot[applied={T|T < 7315338091324563456}]
I20260805 23:06:53.739455 19495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:06:53.739641 19495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:06:53.742766 19495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc66969683064f52a2c93909405fa3fb: Sending DeleteTablet for 3 replicas of tablet 0c6cd955908e4e1fad58c8a969131ead
I20260805 23:06:53.743570 20112 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6cd955908e4e1fad58c8a969131ead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:06:53 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:52398
I20260805 23:06:53.744591 19722 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6cd955908e4e1fad58c8a969131ead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:06:53 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:57756
I20260805 23:06:53.745395 19927 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6cd955908e4e1fad58c8a969131ead with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:06:53 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:35098
I20260805 23:06:53.746805 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 19617
I20260805 23:06:53.750953 20259 tablet_replica.cc:330] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: stopping tablet replica
I20260805 23:06:53.751340 20259 raft_consensus.cc:2229] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:06:53.751672 20259 raft_consensus.cc:2258] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:06:53.755228 20259 ts_tablet_manager.cc:1905] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:06:53.758288 20259 ts_tablet_manager.cc:1918] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:06:53.758406 20259 log.cc:1196] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/0c6cd955908e4e1fad58c8a969131ead
I20260805 23:06:53.758771 20259 ts_tablet_manager.cc:1939] T 0c6cd955908e4e1fad58c8a969131ead P ef142913200d40758ed2f49364e7b7e7: Deleting consensus metadata
I20260805 23:06:53.760227 20262 tablet_replica.cc:330] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: stopping tablet replica
I20260805 23:06:53.760628 20262 raft_consensus.cc:2229] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:06:53.760999 20262 raft_consensus.cc:2258] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:06:53.761382 19475 catalog_manager.cc:4841] TS ef142913200d40758ed2f49364e7b7e7 (127.3.12.67:45273): tablet 0c6cd955908e4e1fad58c8a969131ead (table test-table [id=f5901f7bd2f440079546b92156b2e230]) successfully deleted
I20260805 23:06:53.762691 20262 ts_tablet_manager.cc:1905] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:06:53.765892 20262 ts_tablet_manager.cc:1918] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:06:53.766075 20262 log.cc:1196] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/0c6cd955908e4e1fad58c8a969131ead
I20260805 23:06:53.766556 20262 ts_tablet_manager.cc:1939] T 0c6cd955908e4e1fad58c8a969131ead P c8331e2f7f6b4edd85a068bd5f32b125: Deleting consensus metadata
W20260805 23:06:53.767127 19474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:42671 (error 108)
I20260805 23:06:53.768126 19476 catalog_manager.cc:4841] TS c8331e2f7f6b4edd85a068bd5f32b125 (127.3.12.66:42013): tablet 0c6cd955908e4e1fad58c8a969131ead (table test-table [id=f5901f7bd2f440079546b92156b2e230]) successfully deleted
I20260805 23:06:53.768848 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:06:53.768505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42671 (local address 127.0.0.1:57778)
0805 23:06:53.768687 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0805 23:06:53.768760 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:42671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260805 23:06:53.769163 19474 catalog_manager.cc:4584] TS dd8be7513dd4411dac39b4b938f1373e (127.3.12.65:42671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c6cd955908e4e1fad58c8a969131ead: Network error: Client connection negotiation failed: client connection to 127.3.12.65:42671: connect: Connection refused (error 111)
I20260805 23:06:53.769577 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 19837
I20260805 23:06:53.796624 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 20021
I20260805 23:06:53.816828 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 19456
I20260805 23:06:53.837683 3121 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260805 23:06:54.184451 3121 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260805 23:06:54.184625 3121 mini_ranger.cc:66] Stopping Ranger...
I20260805 23:06:54.540334 3121 mini_ranger.cc:68] Stopped Ranger
2026-08-05 23:06:54.540 UTC [18896] LOG: received smart shutdown request
2026-08-05 23:06:54.545 UTC [18896] LOG: background worker "logical replication launcher" (PID 18910) exited with exit code 1
2026-08-05 23:06:54.545 UTC [18903] LOG: shutting down
2026-08-05 23:06:54.721 UTC [18896] LOG: database system is shut down
2026-08-05T23:06:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50033 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:06:54 dist-test-slave-2d19 krb5kdc[20283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:06:54 dist-test-slave-2d19 krb5kdc[20283](info): set up 2 sockets
Aug 05 23:06:54 dist-test-slave-2d19 krb5kdc[20283](info): commencing operation
krb5kdc: starting...
W20260805 23:06:56.080484 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:06:56 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:06:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/rangeradmin_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/rangeradmin_127.3.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/rangerlookup_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/rangerlookup_127.3.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/HTTP_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/HTTP_127.3.12.84@KRBTEST.COM.keytab.
I20260805 23:06:56.269313 3121 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqJ6txj/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/postgres -l logfile start
2026-08-05 23:06:58.959 UTC [20332] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-05 23:06:58.959 UTC [20332] LOG: listening on IPv4 address "127.3.12.84", port 48117
2026-08-05 23:06:58.963 UTC [20332] LOG: listening on Unix socket "/tmp/.s.PGSQL.48117"
2026-08-05 23:06:58.985 UTC [20335] LOG: database system was shut down at 2026-08-05 23:06:57 UTC
2026-08-05 23:06:58.989 UTC [20332] LOG: database system is ready to accept connections
I20260805 23:07:00.121771 3121 mini_postgres.cc:96] Postgres bound to 48117
2026-08-05 23:07:00.129 UTC [20345] FATAL: database "slave" does not exist
127.3.12.84:48117 - accepting connections
I20260805 23:07:00.130028 3121 mini_ranger.cc:161] Starting Ranger...
I20260805 23:07:00.146828 3121 mini_ranger.cc:84] Created miniranger Postgres user
I20260805 23:07:00.655514 3121 mini_ranger.cc:87] Created ranger Postgres database
I20260805 23:07:00.655606 3121 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-admin
2026-08-05 23:07:00,934 [I] DB FLAVOR :POSTGRES
2026-08-05 23:07:00,934 [I] --------- Verifying Ranger DB connection ---------
2026-08-05 23:07:00,935 [I] Checking connection..
2026-08-05 23:07:00,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-05 23:07:01,176 [I] Checking connection passed.
2026-08-05 23:07:01,176 [I] --------- Verifying version history table ---------
2026-08-05 23:07:01,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 23:07:01,464 [I] Table x_db_version_h does not exist in database ranger
2026-08-05 23:07:01,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 23:07:01,727 [I] Table x_db_version_h does not exist in database ranger
2026-08-05 23:07:01,727 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-05 23:07:01,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-05 23:07:01.987 UTC [20442] WARNING: there is no transaction in progress
2026-08-05 23:07:01,997 [I] create_dbversion_catalog.sql file imported successfully
2026-08-05 23:07:01,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-05 23:07:02,238 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-05 23:07:02,238 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-05 23:07:02,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-05 23:07:02,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-05 23:07:02,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2d19.c.gcp-upstream.internal','N') ;"
2026-08-05 23:07:03,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-05 23:07:03,279 [I] Table x_portal_user does not exist in database ranger
2026-08-05 23:07:03,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-05 23:07:03,552 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-05 23:07:03,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-05 23:07:03,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-05 23:07:04,057 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-05 23:07:04,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-05 23:07:05.079 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.096 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.112 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.128 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.144 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.162 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.217 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.224 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.241 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.252 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.263 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.278 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.288 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.295 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.302 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05.309 UTC [20592] WARNING: there is no transaction in progress
2026-08-05 23:07:05,987 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-05 23:07:05,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-05 23:07:06,256 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-05 23:07:06,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-05 23:07:06,529 [I] Table x_portal_user already exists in database 'ranger'
2026-08-05 23:07:06,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-05 23:07:06,806 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-05 23:07:06,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-05 23:07:07,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-05 23:07:07,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2d19.c.gcp-upstream.internal';"
2026-08-05 23:07:07,620 [I] CORE_DB_SCHEMA import status has been updated
2026-08-05 23:07:07,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-05 23:07:07,885 [I] DB_PATCHES have already been applied
I20260805 23:07:07.891829 3121 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqJ6txj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260805 23:07:07.891914 3121 mini_ranger.cc:193] Using host: 127.3.12.84
I20260805 23:07:07.896548 3121 mini_ranger.cc:227] Ranger admin URL: http://127.3.12.84:38469
Aug 05, 2026 11:07:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 05, 2026 11:07:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 05, 2026 11:07:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-admin/ews/webapp
Aug 05, 2026 11:07:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 05, 2026 11:07:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 05, 2026 11:07:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-admin/ews/webapp].
Aug 05, 2026 11:07:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260805 23:07:09.210129 3121 mini_ranger.cc:160] Time spent starting Ranger: real 9.080s user 0.004s sys 0.008s
Aug 05, 2026 11:07:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 05, 2026 11:07:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 05, 2026 11:07:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 05, 2026 11:07:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 05, 2026 11:07:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260805 23:07:30.524480 3121 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/rangerkms_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/rangerkms_127.3.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.12.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.12.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.12.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/HTTP_127.3.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.12.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/HTTP_127.3.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 05 23:07:30 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971250, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260805 23:07:30.618475 3121 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260805 23:07:30.630218 3121 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260805 23:07:31.538620 3121 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260805 23:07:31.538712 3121 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms
I20260805 23:07:31.538739 3121 mini_ranger_kms.cc:227] Using postgres at 127.3.12.84:48117
2026-08-05 23:07:31,719 [I] DB FLAVOR :POSTGRES
2026-08-05 23:07:31,719 [I] --------- Verifying Ranger DB connection ---------
2026-08-05 23:07:31,720 [I] Checking connection
2026-08-05 23:07:31,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-05 23:07:31,959 [I] connection success
2026-08-05 23:07:31,959 [I] --------- Verifying Ranger DB tables ---------
2026-08-05 23:07:31,959 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-05 23:07:31,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-05 23:07:32,219 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-05 23:07:32,219 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-05 23:07:32,219 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-05 23:07:32,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.12.84:48117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-05 23:07:32,519 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260805 23:07:32.983948 3121 mini_ranger_kms.cc:310] Created kms service
I20260805 23:07:33.177028 3121 mini_ranger_kms.cc:326] Created kudu user
I20260805 23:07:33.224470 3121 mini_ranger_kms.cc:343] Created rangerkms user
Aug 05 23:07:33 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971250, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:33.644511 3121 mini_ranger_kms.cc:384] Added ranger policy
I20260805 23:07:33.644685 3121 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqJ6txj/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqJ6txj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260805 23:07:33.644716 3121 mini_ranger_kms.cc:240] Using host: 127.3.12.84
I20260805 23:07:33.649628 3121 mini_ranger_kms.cc:276] Ranger KMS PID: 20885
I20260805 23:07:33.649724 3121 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.12.84:57737
Aug 05, 2026 11:07:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 05, 2026 11:07:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 05, 2026 11:07:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 05, 2026 11:07:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 05, 2026 11:07:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-kms/ews/webapp].
Aug 05, 2026 11:07:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 05 23:07:36 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785971256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.12.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:36.393316 3121 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.775s user 0.025s sys 0.142s
I20260805 23:07:36.393456 3121 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260805 23:07:36.393559 3121 mini_ranger_kms.cc:401] 127.3.12.84:57737/kms/v1/keys
Aug 05 23:07:36 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.12.84@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
Aug 05 23:07:37 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.12.84@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
Aug 05 23:07:39 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:07:39.427965 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37683
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:37683
--ranger_config_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:57737/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:39.543614 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:39.543928 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:39.544003 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:39.548075 20956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:39.548177 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:39.548216 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:39.548293 20956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:39.548349 20956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:39.553998 20956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.12.84:57737/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:37683
--ranger_config_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:39.555117 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:39.556159 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:39.565157 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:39.565207 20956 server_base.cc:1034] running on GCE node
W20260805 23:07:39.565220 20969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:39.565366 20972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:39.566150 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:39.567267 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:39.568461 20956 hybrid_clock.cc:648] HybridClock initialized: now 1785971259568434 us; error 38 us; skew 500 ppm
Aug 05 23:07:39 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971259, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:39.571215 20956 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:07:39.572157 20956 webserver.cc:460] Webserver started at http://127.3.12.126:37969/ using document root <none> and password file <none>
I20260805 23:07:39.572732 20956 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:39.572817 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:39.573052 20956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 23:07:39 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971259, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:39.628829 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "7473bdc6cdba44a2bb855e490be9b1cc"
format_stamp: "Formatted at 2026-08-05 23:07:39 on dist-test-slave-2d19"
server_key: "4ef5224db5e0d210625a8602f8d33141"
server_key_iv: "03f8352ea9132c31cd7090109cbf1dda"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:39.629475 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "7473bdc6cdba44a2bb855e490be9b1cc"
format_stamp: "Formatted at 2026-08-05 23:07:39 on dist-test-slave-2d19"
server_key: "4ef5224db5e0d210625a8602f8d33141"
server_key_iv: "03f8352ea9132c31cd7090109cbf1dda"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:39.633574 20956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260805 23:07:39.653113 20982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:39.654595 20956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:07:39.654747 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "7473bdc6cdba44a2bb855e490be9b1cc"
format_stamp: "Formatted at 2026-08-05 23:07:39 on dist-test-slave-2d19"
server_key: "4ef5224db5e0d210625a8602f8d33141"
server_key_iv: "03f8352ea9132c31cd7090109cbf1dda"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:39.654892 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:39.679368 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:39.682112 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:39.682327 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:39.690073 20956 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:37683
I20260805 23:07:39.690073 21056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:37683 every 8 connection(s)
I20260805 23:07:39.691138 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:07:39.693965 21057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:39.696374 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 20956
I20260805 23:07:39.696468 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:07:39.696816 3121 external_mini_cluster.cc:1428] Setting key 64df08679fcaf83a4870ac28d2f91b6b
I20260805 23:07:39.700423 21057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Bootstrap starting.
I20260805 23:07:39.703226 21057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:39.704135 21057 log.cc:824] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:39.706555 21057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: No bootstrap required, opened a new log
Aug 05 23:07:39 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:39.710285 21057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } }
I20260805 23:07:39.710584 21057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:39.710654 21057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7473bdc6cdba44a2bb855e490be9b1cc, State: Initialized, Role: FOLLOWER
I20260805 23:07:39.711177 21057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } }
I20260805 23:07:39.711309 21057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:07:39.711388 21057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:07:39.711485 21057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:39.712658 21057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } }
I20260805 23:07:39.713052 21057 leader_election.cc:302] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7473bdc6cdba44a2bb855e490be9b1cc; no voters:
I20260805 23:07:39.713326 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:39.698311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.3.12.126:37683)
0805 23:07:39.698746 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:39.698767 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:39.698844 (+ 77us) server_negotiation.cc:407] Connection header received
0805 23:07:39.699837 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:39.699863 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:39.700383 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:39.700771 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:39.701820 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:39.702947 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:39.704182 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:39.704851 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:39.705034 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:39.708099 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:39.708104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:39.708118 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:39.708143 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:39.710086 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:39.710626 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:39.710638 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:39.710659 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:39.710801 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:39.711111 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:39.711114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:39.711115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:39.711650 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:39.711805 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:39.712134 (+ 329us) server_negotiation.cc:299] Negotiation successful
0805 23:07:39.712446 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":159,"threads_started":1}
I20260805 23:07:39.713382 21057 leader_election.cc:288] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:07:39.713508 21065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:07:39.713758 21065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 1 LEADER]: Becoming Leader. State: Replica: 7473bdc6cdba44a2bb855e490be9b1cc, State: Running, Role: LEADER
I20260805 23:07:39.714138 21065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } }
I20260805 23:07:39.714659 21057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:07:39.715400 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7473bdc6cdba44a2bb855e490be9b1cc. Latest consensus state: current_term: 1 leader_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } } }
I20260805 23:07:39.715526 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:39.715883 21068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } } }
I20260805 23:07:39.715969 21068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:39.716902 21057 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260805 23:07:39.717974 21077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:07:39.724068 21077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:07:39.730945 21077 catalog_manager.cc:1347] Generated new cluster ID: d1dfeccd0fed4a50b38cf93c54a8afd6
I20260805 23:07:39.731017 21077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:07:39.759281 21077 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:07:39.760368 21077 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:07:39.771166 21077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Generated new TSK 0
I20260805 23:07:39.771865 21077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:07:40.431279 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:57737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37683
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:40.594607 21117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:40.594900 21117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:40.594993 21117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:40.599236 21117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:40.599323 21117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:40.599429 21117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:07:40.604889 21117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.12.84:57737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:40.606007 21117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:40.607069 21117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:40.617767 21130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:40.617767 21128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:40.618075 21126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:40.618381 21117 server_base.cc:1034] running on GCE node
I20260805 23:07:40.618939 21117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:40.619650 21117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:40.620829 21117 hybrid_clock.cc:648] HybridClock initialized: now 1785971260620793 us; error 63 us; skew 500 ppm
Aug 05 23:07:40 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:40.623790 21117 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:07:40.624884 21117 webserver.cc:460] Webserver started at http://127.3.12.65:35291/ using document root <none> and password file <none>
I20260805 23:07:40.625474 21117 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:40.625571 21117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:40.625794 21117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 23:07:40 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:40.639696 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "9bae81f907b84a209adb07aa4d14bac2"
format_stamp: "Formatted at 2026-08-05 23:07:40 on dist-test-slave-2d19"
server_key: "9e3b22006774aa50d56f1e546acf470d"
server_key_iv: "f8a9745b1ca8e18a79477088eb94eea9"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:40.640556 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "9bae81f907b84a209adb07aa4d14bac2"
format_stamp: "Formatted at 2026-08-05 23:07:40 on dist-test-slave-2d19"
server_key: "9e3b22006774aa50d56f1e546acf470d"
server_key_iv: "f8a9745b1ca8e18a79477088eb94eea9"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:40.646126 21117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260805 23:07:40.673281 21142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:40.675330 21117 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260805 23:07:40.675491 21117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "9bae81f907b84a209adb07aa4d14bac2"
format_stamp: "Formatted at 2026-08-05 23:07:40 on dist-test-slave-2d19"
server_key: "9e3b22006774aa50d56f1e546acf470d"
server_key_iv: "f8a9745b1ca8e18a79477088eb94eea9"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:40.675669 21117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:40.704707 21117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:40.708750 21117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:40.709030 21117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:40.709854 21117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:40.711055 21117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:40.711175 21117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:40.711323 21117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:40.711439 21117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:40.725533 21117 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:44997
I20260805 23:07:40.726852 21117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:07:40.729733 21305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:44997 every 8 connection(s)
I20260805 23:07:40.733083 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 21117
I20260805 23:07:40.733208 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:07:40.733523 3121 external_mini_cluster.cc:1428] Setting key b411082a4d5e807aff45347e40e56d27
Aug 05 23:07:40 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:07:40.752128 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:40.733522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37683 (local address 127.3.12.65:58763)
0805 23:07:40.735733 (+ 2211us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:40.735793 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:40.736927 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:40.737764 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:40.737773 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:40.738139 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:40.738832 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:40.738852 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:40.740179 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:40.740183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:40.741832 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:40.741845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:40.742657 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:40.742662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:40.742883 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:40.743688 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:40.743712 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:40.745697 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:40.748331 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:40.748339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:40.748357 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:40.748743 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:40.749117 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:40.749121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:40.749123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:40.749316 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:40.749902 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:40.749910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:40.750325 (+ 415us) client_negotiation.cc:769] Sending connection context
0805 23:07:40.750545 (+ 220us) client_negotiation.cc:240] Negotiation successful
0805 23:07:40.750918 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"thread_start_us":170,"threads_started":1}
I20260805 23:07:40.752449 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:40.735025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:58763 (local address 127.3.12.126:37683)
0805 23:07:40.735643 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:40.735653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:40.735923 (+ 270us) server_negotiation.cc:407] Connection header received
0805 23:07:40.737294 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:40.737298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:40.737353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:40.737441 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:40.739013 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:40.739977 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:40.742015 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:40.742451 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:40.743280 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:40.745915 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:40.745919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:40.745922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:40.745952 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:40.748141 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:40.748887 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:40.748891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:40.748893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:40.748988 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:40.749502 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:40.749505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:40.749507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:40.749759 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:40.751940 (+ 2181us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:40.752097 (+ 157us) server_negotiation.cc:299] Negotiation successful
0805 23:07:40.752242 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":146,"threads_started":1}
I20260805 23:07:40.756328 21306 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37683
I20260805 23:07:40.756626 21306 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:40.757268 21306 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:40.759215 21004 ts_manager.cc:194] Registered new tserver with Master: 9bae81f907b84a209adb07aa4d14bac2 (127.3.12.65:44997)
I20260805 23:07:40.760784 21004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:58763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:07:40.783871 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:57737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37683
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 23:07:40 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785971260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260805 23:07:40.953325 21313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:40.953742 21313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:40.953946 21313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:40.959925 21313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:40.960009 21313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:40.960126 21313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:07:40.966851 21313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.12.84:57737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:40.968226 21313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:40.969552 21313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:40.980176 21320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:40.980356 21325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:40.980201 21319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:40.983469 21313 server_base.cc:1034] running on GCE node
I20260805 23:07:40.983970 21313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:40.984936 21313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:40.986153 21313 hybrid_clock.cc:648] HybridClock initialized: now 1785971260986119 us; error 53 us; skew 500 ppm
Aug 05 23:07:40 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:40.990310 21313 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:07:40.991601 21313 webserver.cc:460] Webserver started at http://127.3.12.66:41213/ using document root <none> and password file <none>
I20260805 23:07:40.992446 21313 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:40.992599 21313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:40.992920 21313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 23:07:40 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:41.016095 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "39ce7ca484e14aaaa3dfc2be2009da66"
format_stamp: "Formatted at 2026-08-05 23:07:41 on dist-test-slave-2d19"
server_key: "9cf161234232cea2244a0cefe75f9bb5"
server_key_iv: "fcd47541acb8dcd603d17403ff9d41cb"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:41.018294 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "39ce7ca484e14aaaa3dfc2be2009da66"
format_stamp: "Formatted at 2026-08-05 23:07:41 on dist-test-slave-2d19"
server_key: "9cf161234232cea2244a0cefe75f9bb5"
server_key_iv: "fcd47541acb8dcd603d17403ff9d41cb"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:41.024133 21313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260805 23:07:41.044692 21334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:41.046458 21313 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260805 23:07:41.046684 21313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "39ce7ca484e14aaaa3dfc2be2009da66"
format_stamp: "Formatted at 2026-08-05 23:07:41 on dist-test-slave-2d19"
server_key: "9cf161234232cea2244a0cefe75f9bb5"
server_key_iv: "fcd47541acb8dcd603d17403ff9d41cb"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:41.046919 21313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:41.077627 21313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:41.081488 21313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:41.081808 21313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:41.082544 21313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:41.083731 21313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:41.083840 21313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:41.083940 21313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:41.084007 21313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:41.102836 21313 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:41847
I20260805 23:07:41.104122 21313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:07:41.105078 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 21313
I20260805 23:07:41.105191 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:07:41.105511 3121 external_mini_cluster.cc:1428] Setting key b6db4b096818e4880e6026c5cd75b19f
I20260805 23:07:41.113039 21500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:41847 every 8 connection(s)
Aug 05 23:07:41 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:41.129738 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:41.111305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:47277 (local address 127.3.12.126:37683)
0805 23:07:41.111439 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:41.111444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:41.113648 (+ 2204us) server_negotiation.cc:407] Connection header received
0805 23:07:41.115620 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:41.115631 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:41.115718 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:41.115823 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:41.118315 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:41.119189 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:41.121005 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:41.121380 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:41.121479 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:41.125008 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:41.125012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:41.125015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:41.125047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:41.126915 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:41.127704 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:41.127708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:41.127710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:41.127811 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:41.128295 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:41.128300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:41.128302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:41.128558 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:41.128649 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:41.129393 (+ 744us) server_negotiation.cc:299] Negotiation successful
0805 23:07:41.129556 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:07:41.131711 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:41.111701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37683 (local address 127.3.12.66:47277)
0805 23:07:41.113486 (+ 1785us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:41.113551 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:41.114668 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:41.116539 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:41.116549 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:41.116925 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:41.117631 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:41.117653 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:41.119363 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:41.119366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:41.120844 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:41.120859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:41.121717 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:41.121722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:41.121927 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:41.122761 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:41.122783 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:41.124791 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:41.127095 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:41.127102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:41.127123 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:41.127544 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:41.127938 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:41.127941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:41.127945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:41.128149 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:41.128731 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:41.128738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:41.129100 (+ 362us) client_negotiation.cc:769] Sending connection context
0805 23:07:41.130211 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0805 23:07:41.130593 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1563,"thread_start_us":197,"threads_started":1}
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:07:41.133347 21501 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37683
I20260805 23:07:41.133658 21501 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:41.134317 21501 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:41.135651 21004 ts_manager.cc:194] Registered new tserver with Master: 39ce7ca484e14aaaa3dfc2be2009da66 (127.3.12.66:41847)
I20260805 23:07:41.136439 21004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:47277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:07:41.171931 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:57737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37683
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:41.369462 21507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:41.369952 21507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:41.370076 21507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:41.376133 21507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:41.376294 21507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:41.376436 21507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:07:41.384407 21507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.12.84:57737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:41.386080 21507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:41.387523 21507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:41.403064 21518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:41.403398 21507 server_base.cc:1034] running on GCE node
W20260805 23:07:41.403515 21516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:41.415877 21514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:41.416672 21507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:41.417660 21507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:41.418877 21507 hybrid_clock.cc:648] HybridClock initialized: now 1785971261418801 us; error 93 us; skew 500 ppm
Aug 05 23:07:41 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
Aug 05 23:07:41 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:41.423158 21507 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:07:41.424377 21507 webserver.cc:460] Webserver started at http://127.3.12.67:35599/ using document root <none> and password file <none>
I20260805 23:07:41.425155 21507 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:41.425246 21507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:41.425477 21507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 05 23:07:41 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:41.448969 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "008496c81e3d47719fab994ff681d069"
format_stamp: "Formatted at 2026-08-05 23:07:41 on dist-test-slave-2d19"
server_key: "d54ccc2c46c82ebd23294fa3295eeefe"
server_key_iv: "32614a65844827b074216c6508c97850"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:41.450105 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "008496c81e3d47719fab994ff681d069"
format_stamp: "Formatted at 2026-08-05 23:07:41 on dist-test-slave-2d19"
server_key: "d54ccc2c46c82ebd23294fa3295eeefe"
server_key_iv: "32614a65844827b074216c6508c97850"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:41.457809 21507 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260805 23:07:41.482568 21529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:41.484485 21507 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260805 23:07:41.484632 21507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "008496c81e3d47719fab994ff681d069"
format_stamp: "Formatted at 2026-08-05 23:07:41 on dist-test-slave-2d19"
server_key: "d54ccc2c46c82ebd23294fa3295eeefe"
server_key_iv: "32614a65844827b074216c6508c97850"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:41.484768 21507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:41.499626 21507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:41.503615 21507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:41.503883 21507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:41.504688 21507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:41.505829 21507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:41.505944 21507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:41.506060 21507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:41.506129 21507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:41.521966 21507 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:45439
I20260805 23:07:41.523355 21507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:07:41.523859 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 21507
I20260805 23:07:41.523990 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:07:41.524344 3121 external_mini_cluster.cc:1428] Setting key ff66e6066ce20497090365890374c4d4
I20260805 23:07:41.529326 21688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:45439 every 8 connection(s)
Aug 05 23:07:41 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:41.544679 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:41.528653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:52211 (local address 127.3.12.126:37683)
0805 23:07:41.528787 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:41.528791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:41.529914 (+ 1123us) server_negotiation.cc:407] Connection header received
0805 23:07:41.531203 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:41.531207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:41.531263 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:41.531316 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:41.533178 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:41.534029 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:41.535865 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:41.536239 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:41.536355 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:41.539862 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:41.539866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:41.539870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:41.539899 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:41.541963 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:41.542727 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:41.542731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:41.542733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:41.542827 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:41.543258 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:41.543261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:41.543263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:41.543501 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:41.543572 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:41.544353 (+ 781us) server_negotiation.cc:299] Negotiation successful
0805 23:07:41.544508 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:07:41.546731 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:41.527453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37683 (local address 127.3.12.67:52211)
0805 23:07:41.529748 (+ 2295us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:41.529807 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:41.530935 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:41.531751 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:41.531761 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:41.532174 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:41.532911 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:41.532932 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:41.534197 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:41.534200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:41.535717 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:41.535728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:41.536721 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:41.536725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:41.537009 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:41.537769 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:41.537790 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:41.539668 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:41.542143 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:41.542150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:41.542170 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:41.542599 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:41.542940 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:41.542943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:41.542945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:41.543142 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:41.543656 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:41.543663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:41.544028 (+ 365us) client_negotiation.cc:769] Sending connection context
0805 23:07:41.545199 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0805 23:07:41.545619 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":171,"threads_started":1}
I20260805 23:07:41.548461 21689 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37683
I20260805 23:07:41.548715 21689 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:41.549322 21689 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:41.551654 21004 ts_manager.cc:194] Registered new tserver with Master: 008496c81e3d47719fab994ff681d069 (127.3.12.67:45439)
I20260805 23:07:41.552423 21004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:52211
I20260805 23:07:41.563016 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:07:41.564000 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 21117
I20260805 23:07:41.592959 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 21313
I20260805 23:07:41.603538 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 21507
I20260805 23:07:41.655635 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 20956
I20260805 23:07:41.677632 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37683
--webserver_interface=127.3.12.126
--webserver_port=37969
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:37683
--ranger_config_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:57737/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.12.84:57737/kms with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:41.800823 21694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:41.801107 21694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:41.801149 21694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:41.805097 21694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:41.805159 21694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:41.805179 21694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:41.805195 21694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:41.805210 21694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:41.810686 21694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.12.84:57737/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:37683
--ranger_config_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=37969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:41.811755 21694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:41.812770 21694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:41.821313 21699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:41.821322 21702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:41.821377 21694 server_base.cc:1034] running on GCE node
W20260805 23:07:41.821331 21700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:41.822170 21694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:41.823263 21694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:41.824468 21694 hybrid_clock.cc:648] HybridClock initialized: now 1785971261824432 us; error 49 us; skew 500 ppm
Aug 05 23:07:41 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:41.827395 21694 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:07:41.828547 21694 webserver.cc:460] Webserver started at http://127.3.12.126:37969/ using document root <none> and password file <none>
I20260805 23:07:41.829150 21694 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:41.829254 21694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 05 23:07:41 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:41.885252 21694 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260805 23:07:41.887562 21714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:41.888911 21694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:07:41.889063 21694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "7473bdc6cdba44a2bb855e490be9b1cc"
format_stamp: "Formatted at 2026-08-05 23:07:39 on dist-test-slave-2d19"
server_key: "4ef5224db5e0d210625a8602f8d33141"
server_key_iv: "03f8352ea9132c31cd7090109cbf1dda"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:41.889655 21694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:41.910359 21694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:41.913632 21694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:41.913856 21694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:41.921317 21694 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:37683
I20260805 23:07:41.921372 21793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:37683 every 8 connection(s)
I20260805 23:07:41.922477 21694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:07:41.926896 21794 sys_catalog.cc:263] Verifying existing consensus state
I20260805 23:07:41.928056 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 21694
I20260805 23:07:41.928568 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:57737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:44997
--local_ip_for_outbound_sockets=127.3.12.65
--tserver_master_addrs=127.3.12.126:37683
--webserver_port=35291
--webserver_interface=127.3.12.65
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.12.84:57737/kms with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:07:41.928664 21794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Bootstrap starting.
I20260805 23:07:41.945664 21794 log.cc:824] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:41.952600 21794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:07:41.953018 21794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Bootstrap complete.
I20260805 23:07:41.957391 21794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } }
I20260805 23:07:41.957813 21794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7473bdc6cdba44a2bb855e490be9b1cc, State: Initialized, Role: FOLLOWER
I20260805 23:07:41.958353 21794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } }
I20260805 23:07:41.958498 21794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:07:41.958591 21794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:07:41.958707 21794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 1 FOLLOWER]: Advancing to term 2
I20260805 23:07:41.960700 21794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } }
I20260805 23:07:41.961036 21794 leader_election.cc:302] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7473bdc6cdba44a2bb855e490be9b1cc; no voters:
I20260805 23:07:41.961366 21794 leader_election.cc:288] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260805 23:07:41.961539 21799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 2 FOLLOWER]: Leader election won for term 2
I20260805 23:07:41.961869 21799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [term 2 LEADER]: Becoming Leader. State: Replica: 7473bdc6cdba44a2bb855e490be9b1cc, State: Running, Role: LEADER
I20260805 23:07:41.962245 21799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } }
I20260805 23:07:41.962625 21794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:07:41.963800 21800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } } }
I20260805 23:07:41.963925 21800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:41.964399 21801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7473bdc6cdba44a2bb855e490be9b1cc. Latest consensus state: current_term: 2 leader_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7473bdc6cdba44a2bb855e490be9b1cc" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37683 } } }
I20260805 23:07:41.964563 21801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:41.964941 21810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:07:41.965103 21794 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260805 23:07:41.971413 21810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:07:41.972394 21810 catalog_manager.cc:1259] Loaded cluster ID: d1dfeccd0fed4a50b38cf93c54a8afd6
I20260805 23:07:41.972512 21810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:07:41.975723 21810 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:07:41.976513 21810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Loaded TSK: 0
I20260805 23:07:41.977990 21810 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 23:07:42.083907 21797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:42.084507 21797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:42.084611 21797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:42.091277 21797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:42.091363 21797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:42.091454 21797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:07:42.099730 21797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.12.84:57737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:44997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=35291
--tserver_master_addrs=127.3.12.126:37683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:42.101413 21797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:42.102881 21797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:42.116518 21843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:42.116997 21846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:42.117717 21842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:42.118563 21797 server_base.cc:1034] running on GCE node
I20260805 23:07:42.119197 21797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:42.120049 21797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:42.121241 21797 hybrid_clock.cc:648] HybridClock initialized: now 1785971262121174 us; error 109 us; skew 500 ppm
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:42.124845 21797 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:07:42.126035 21797 webserver.cc:460] Webserver started at http://127.3.12.65:35291/ using document root <none> and password file <none>
I20260805 23:07:42.126806 21797 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:42.126914 21797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:42.148293 21797 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 23:07:42.151918 21857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.153609 21797 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260805 23:07:42.153738 21797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "9bae81f907b84a209adb07aa4d14bac2"
format_stamp: "Formatted at 2026-08-05 23:07:40 on dist-test-slave-2d19"
server_key: "9e3b22006774aa50d56f1e546acf470d"
server_key_iv: "f8a9745b1ca8e18a79477088eb94eea9"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:42.154459 21797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:42.170220 21797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:42.174152 21797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:42.174378 21797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:42.175120 21797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:42.176203 21797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:42.176287 21797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.176345 21797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:42.176417 21797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.188087 21797 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:44997
I20260805 23:07:42.188274 22021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:44997 every 8 connection(s)
I20260805 23:07:42.189263 21797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:07:42.198874 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 21797
I20260805 23:07:42.199431 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:57737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:41847
--local_ip_for_outbound_sockets=127.3.12.66
--tserver_master_addrs=127.3.12.126:37683
--webserver_port=41213
--webserver_interface=127.3.12.66
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.12.84:57737/kms with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:42.212922 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:42.192003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37683 (local address 127.3.12.65:41429)
0805 23:07:42.194640 (+ 2637us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:42.194697 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:42.195766 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:42.197318 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:42.197328 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:42.197712 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:42.198393 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:42.198415 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.199710 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.199715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:42.200736 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:42.200746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.201992 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.201998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:42.202214 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:42.204464 (+ 2250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:42.204487 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:42.206277 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:42.209096 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:42.209103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:42.209118 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:42.209478 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:42.209900 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:42.209904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:42.209906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:42.210078 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:42.210877 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:42.210884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:42.211232 (+ 348us) client_negotiation.cc:769] Sending connection context
0805 23:07:42.211451 (+ 219us) client_negotiation.cc:240] Negotiation successful
0805 23:07:42.211796 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2422,"thread_start_us":1170,"threads_started":1}
I20260805 23:07:42.214519 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:42.192797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:41429 (local address 127.3.12.126:37683)
0805 23:07:42.194216 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:42.194238 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:42.196102 (+ 1864us) server_negotiation.cc:407] Connection header received
0805 23:07:42.196422 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:42.196436 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:42.196824 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:42.197177 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:42.198609 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.199564 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.200876 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.201559 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.201733 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:42.206543 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:42.206553 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:42.206574 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:42.206614 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:42.208945 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:42.209629 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:42.209636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:42.209641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:42.209785 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:42.210192 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:42.210195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:42.210197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:42.210684 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:42.210819 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:42.213301 (+ 2482us) server_negotiation.cc:299] Negotiation successful
0805 23:07:42.213615 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":137,"threads_started":1}
I20260805 23:07:42.217500 22023 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37683
I20260805 23:07:42.217741 22023 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.218305 22023 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
W20260805 23:07:42.220415 22023 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:37683 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 23:07:42.220556 22023 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.220738 22023 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:42.221735 22023 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.221905 22023 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
W20260805 23:07:42.222844 22023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 23:07:42.357419 22030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:42.357715 22030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:42.357820 22030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:42.362893 22030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:42.363009 22030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:42.363112 22030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:07:42.370500 22030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.12.84:57737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:41847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=41213
--tserver_master_addrs=127.3.12.126:37683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:42.371649 22030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:42.372828 22030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:42.383939 22040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:42.384465 22042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:42.384486 22030 server_base.cc:1034] running on GCE node
W20260805 23:07:42.384740 22038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:42.385277 22030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:42.386052 22030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:42.387290 22030 hybrid_clock.cc:648] HybridClock initialized: now 1785971262387263 us; error 41 us; skew 500 ppm
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:42.390995 22030 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:07:42.392129 22030 webserver.cc:460] Webserver started at http://127.3.12.66:41213/ using document root <none> and password file <none>
I20260805 23:07:42.392869 22030 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:42.392990 22030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:42.417523 22030 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260805 23:07:42.421017 22053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.422731 22030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260805 23:07:42.422865 22030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "39ce7ca484e14aaaa3dfc2be2009da66"
format_stamp: "Formatted at 2026-08-05 23:07:41 on dist-test-slave-2d19"
server_key: "9cf161234232cea2244a0cefe75f9bb5"
server_key_iv: "fcd47541acb8dcd603d17403ff9d41cb"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:42.423616 22030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:42.449349 22030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:42.455314 22030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:42.455610 22030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:42.456414 22030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:42.457554 22030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:42.457659 22030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.457755 22030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:42.457811 22030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.472306 22030 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:41847
I20260805 23:07:42.474141 22030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:07:42.479839 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 22030
I20260805 23:07:42.480381 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.12.84:57737/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:45439
--local_ip_for_outbound_sockets=127.3.12.67
--tserver_master_addrs=127.3.12.126:37683
--webserver_port=35599
--webserver_interface=127.3.12.67
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.12.84:57737/kms with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:07:42.480386 22222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:41847 every 8 connection(s)
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:42.508823 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:42.484002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37683 (local address 127.3.12.66:38613)
0805 23:07:42.487239 (+ 3237us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:42.487302 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:42.488493 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:42.489356 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:42.489366 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:42.489739 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:42.490428 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:42.490447 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.491616 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.491619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:42.493067 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:42.493174 (+ 107us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.494006 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.494010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:42.494304 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:42.500626 (+ 6322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:42.500655 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:42.502410 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:42.504885 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:42.504892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:42.504912 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:42.505355 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:42.505754 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:42.505757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:42.505759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:42.505975 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:42.506553 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:42.506560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:42.506979 (+ 419us) client_negotiation.cc:769] Sending connection context
0805 23:07:42.507220 (+ 241us) client_negotiation.cc:240] Negotiation successful
0805 23:07:42.507631 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2970,"thread_start_us":173,"threads_started":1}
I20260805 23:07:42.509570 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:42.485231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:38613 (local address 127.3.12.126:37683)
0805 23:07:42.485407 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:42.485413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:42.487427 (+ 2014us) server_negotiation.cc:407] Connection header received
0805 23:07:42.488774 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:42.488778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:42.488841 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:42.488899 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:42.490614 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.491444 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.493297 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.493675 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.493755 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:42.502658 (+ 8903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:42.502663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:42.502666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:42.502703 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:42.504714 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:42.505503 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:42.505508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:42.505509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:42.505626 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:42.506107 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:42.506111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:42.506113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:42.506376 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:42.506486 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:42.509271 (+ 2785us) server_negotiation.cc:299] Negotiation successful
0805 23:07:42.509412 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 23:07:42.511435 22223 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37683
I20260805 23:07:42.511718 22223 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.512398 22223 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
W20260805 23:07:42.514292 22223 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:37683 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 23:07:42.514417 22223 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.514604 22223 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:42.515520 22223 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.515710 22223 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
W20260805 23:07:42.516610 22223 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 23:07:42.638095 22227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:42.638505 22227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:42.638633 22227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:42.644462 22227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:42.644652 22227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:42.644816 22227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:07:42.650899 22227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.12.84:57737/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:45439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=35599
--tserver_master_addrs=127.3.12.126:37683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:42.652088 22227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:42.653214 22227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:42.666458 22241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:42.667066 22227 server_base.cc:1034] running on GCE node
W20260805 23:07:42.668421 22235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:42.668713 22236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:42.669176 22227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:42.669945 22227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:42.673233 22227 hybrid_clock.cc:648] HybridClock initialized: now 1785971262673132 us; error 112 us; skew 500 ppm
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:42.676544 22227 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:07:42.677625 22227 webserver.cc:460] Webserver started at http://127.3.12.67:35599/ using document root <none> and password file <none>
I20260805 23:07:42.678400 22227 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:42.678500 22227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:42.701977 22227 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:07:42.706143 22250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.707893 22227 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260805 23:07:42.708207 22227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "008496c81e3d47719fab994ff681d069"
format_stamp: "Formatted at 2026-08-05 23:07:41 on dist-test-slave-2d19"
server_key: "d54ccc2c46c82ebd23294fa3295eeefe"
server_key_iv: "32614a65844827b074216c6508c97850"
server_key_version: "kuduclusterkey@0"
I20260805 23:07:42.709080 22227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:42.726658 22227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:42.729966 22227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:42.730206 22227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:42.731011 22227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:42.732020 22227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:42.732128 22227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.732210 22227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:42.732240 22227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:42.744119 22227 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:45439
I20260805 23:07:42.745294 22227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:07:42.746083 22423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:45439 every 8 connection(s)
I20260805 23:07:42.748996 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 22227
Aug 05 23:07:42 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971262, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:42.761396 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:42.747439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:56965 (local address 127.3.12.126:37683)
0805 23:07:42.747576 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:42.747581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:42.748487 (+ 906us) server_negotiation.cc:407] Connection header received
0805 23:07:42.750013 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:42.750016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:42.750076 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:42.750184 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:42.751381 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.751920 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.753291 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.753565 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.753661 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:42.756794 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:42.756797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:42.756800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:42.756826 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:42.758631 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:42.759391 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:42.759395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:42.759397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:42.759498 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:42.759928 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:42.759932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:42.759934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:42.760161 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:42.760244 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:42.761036 (+ 792us) server_negotiation.cc:299] Negotiation successful
0805 23:07:42.761217 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260805 23:07:42.762794 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:42.747819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37683 (local address 127.3.12.67:56965)
0805 23:07:42.748319 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:42.748377 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:42.749451 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:42.750372 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:42.750379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:42.750654 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:42.751230 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:42.751245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.752058 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.752062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:42.753164 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:42.753174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.753824 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.753828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:42.754024 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:42.754698 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:42.754720 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:42.756619 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:42.758771 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:42.758777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:42.758793 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:42.759267 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:42.759615 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:42.759618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:42.759620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:42.759810 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:42.760389 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:42.760396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:42.760735 (+ 339us) client_negotiation.cc:769] Sending connection context
0805 23:07:42.761658 (+ 923us) client_negotiation.cc:240] Negotiation successful
0805 23:07:42.761963 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":134,"threads_started":1}
I20260805 23:07:42.764146 22424 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37683
I20260805 23:07:42.764462 22424 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.764992 22424 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:42.765062 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:42.749691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.3.12.126:37683)
0805 23:07:42.751881 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:42.751886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:42.752242 (+ 356us) server_negotiation.cc:407] Connection header received
0805 23:07:42.752404 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:42.752407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:42.752464 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:42.752558 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:42.753532 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.754707 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.756589 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:42.756957 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:42.757406 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:42.760381 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:42.760385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:42.760388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:42.760415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:42.762415 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:42.762980 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:42.762984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:42.762986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:42.763113 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:42.763501 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:42.763505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:42.763507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:42.763725 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:42.763932 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:42.764763 (+ 831us) server_negotiation.cc:299] Negotiation successful
0805 23:07:42.764892 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2110,"thread_start_us":94,"threads_started":1}
W20260805 23:07:42.766657 22424 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:37683 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260805 23:07:42.766812 22424 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.767014 22424 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:42.767925 22424 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:42.768132 22424 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
W20260805 23:07:42.768996 22424 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 05 23:07:43 dist-test-slave-2d19 krb5kdc[20283](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785971263, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:43.223160 22023 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:43.223500 22023 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:43.224911 21741 ts_manager.cc:194] Registered new tserver with Master: 9bae81f907b84a209adb07aa4d14bac2 (127.3.12.65:44997)
I20260805 23:07:43.226547 21741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:41429
I20260805 23:07:43.516883 22223 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:43.517233 22223 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:43.518193 21741 ts_manager.cc:194] Registered new tserver with Master: 39ce7ca484e14aaaa3dfc2be2009da66 (127.3.12.66:41847)
I20260805 23:07:43.519006 21741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:38613
Aug 05 23:07:43 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971263, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:43.769268 22424 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:43.769600 22424 heartbeater.cc:511] Master 127.3.12.126:37683 requested a full tablet report, sending...
I20260805 23:07:43.770506 21741 ts_manager.cc:194] Registered new tserver with Master: 008496c81e3d47719fab994ff681d069 (127.3.12.67:45439)
I20260805 23:07:43.771296 21741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:56965
I20260805 23:07:43.777698 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:07:43.790746 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.779583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.3.12.126:37683)
0805 23:07:43.780285 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:43.780290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:43.780783 (+ 493us) server_negotiation.cc:407] Connection header received
0805 23:07:43.780830 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:43.780833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:43.780882 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:43.780965 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:43.781959 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.782817 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.784050 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.784423 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.784540 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.786435 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:43.786440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:43.786442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:43.786485 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:43.788408 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:43.788943 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:43.788947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:43.788949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:43.789047 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:43.789345 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:43.789348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:43.789350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:43.789585 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:43.789712 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:43.790393 (+ 681us) server_negotiation.cc:299] Negotiation successful
0805 23:07:43.790561 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":89,"threads_started":1}
I20260805 23:07:43.795609 21741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:07:43.800803 21741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:07:43.830133 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.819492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:44997 (local address 127.0.0.1:49978)
0805 23:07:43.820961 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:43.821024 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:43.821179 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:43.824128 (+ 2949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:43.824135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:43.824161 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:43.824324 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.824337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.825941 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.825944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.828116 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.828129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.829528 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.829532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.829653 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.829699 (+ 46us) client_negotiation.cc:769] Sending connection context
0805 23:07:43.829821 (+ 122us) client_negotiation.cc:240] Negotiation successful
0805 23:07:43.829955 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"spinlock_wait_cycles":8320,"thread_start_us":97,"threads_started":1}
I20260805 23:07:43.831846 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.819647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.3.12.65:44997)
0805 23:07:43.822752 (+ 3105us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:43.822760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:43.822785 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:07:43.822865 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:43.822870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:43.823074 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:43.823220 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:43.824579 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.825798 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.828295 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.829170 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.829287 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.829444 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:43.831592 (+ 2148us) server_negotiation.cc:299] Negotiation successful
0805 23:07:43.831702 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2991,"thread_start_us":81,"threads_started":1}
I20260805 23:07:43.835655 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.818026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.3.12.67:45439)
0805 23:07:43.821566 (+ 3540us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:43.821574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:43.821601 (+ 27us) server_negotiation.cc:407] Connection header received
0805 23:07:43.821680 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:43.821685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:43.821915 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:43.822090 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:43.828610 (+ 6520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.829782 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.833199 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.834070 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.834154 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.834293 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:43.835401 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0805 23:07:43.835509 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3399,"thread_start_us":80,"threads_started":1}
I20260805 23:07:43.835351 21933 tablet_service.cc:1467] Processing CreateTablet for tablet 568c73267a284720a32430fcd0d5bd73 (DEFAULT_TABLE table=test-table [id=d48045ae118c4118807190acb677c6b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:43.836341 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.816890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:45439 (local address 127.0.0.1:57012)
0805 23:07:43.820961 (+ 4071us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:43.821024 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:43.821179 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:43.828325 (+ 7146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:43.828328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:43.828348 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:43.828452 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.828458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.829948 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.829952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.833042 (+ 3090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.833054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.834392 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.834395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.834486 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.834508 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:07:43.836092 (+ 1584us) client_negotiation.cc:240] Negotiation successful
0805 23:07:43.836175 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3853,"spinlock_wait_cycles":2304,"thread_start_us":104,"threads_started":1}
I20260805 23:07:43.836619 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c73267a284720a32430fcd0d5bd73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:43.839449 22300 tablet_service.cc:1467] Processing CreateTablet for tablet 568c73267a284720a32430fcd0d5bd73 (DEFAULT_TABLE table=test-table [id=d48045ae118c4118807190acb677c6b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:43.840595 22300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c73267a284720a32430fcd0d5bd73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:43.851460 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.820182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:41847 (local address 127.0.0.1:56166)
0805 23:07:43.822376 (+ 2194us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:43.822391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:43.822498 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:43.831788 (+ 9290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:43.831792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:43.831811 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:43.831893 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.831921 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.848459 (+ 16538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.848464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.849729 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.849744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.851010 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.851015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.851118 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.851152 (+ 34us) client_negotiation.cc:769] Sending connection context
0805 23:07:43.851209 (+ 57us) client_negotiation.cc:240] Negotiation successful
0805 23:07:43.851294 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2126,"thread_start_us":122,"threads_started":1}
I20260805 23:07:43.852640 22476 tablet_bootstrap.cc:492] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2: Bootstrap starting.
I20260805 23:07:43.853353 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.820379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.3.12.66:41847)
0805 23:07:43.830353 (+ 9974us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:43.830360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:43.830384 (+ 24us) server_negotiation.cc:407] Connection header received
0805 23:07:43.830459 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:43.830465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:43.830680 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:43.832094 (+ 1414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:43.832271 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.835234 (+ 2963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.849947 (+ 14713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.850834 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.852008 (+ 1174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.852173 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:43.852230 (+ 57us) server_negotiation.cc:299] Negotiation successful
0805 23:07:43.853191 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9850,"thread_start_us":114,"threads_started":1}
I20260805 23:07:43.856616 22101 tablet_service.cc:1467] Processing CreateTablet for tablet 568c73267a284720a32430fcd0d5bd73 (DEFAULT_TABLE table=test-table [id=d48045ae118c4118807190acb677c6b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:43.857760 22101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568c73267a284720a32430fcd0d5bd73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:43.858248 22476 tablet_bootstrap.cc:654] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:43.859202 22476 log.cc:824] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:43.862095 22476 tablet_bootstrap.cc:492] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2: No bootstrap required, opened a new log
I20260805 23:07:43.862308 22476 ts_tablet_manager.cc:1397] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260805 23:07:43.866835 22476 raft_consensus.cc:348] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.867250 22476 raft_consensus.cc:374] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:43.867359 22476 raft_consensus.cc:729] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bae81f907b84a209adb07aa4d14bac2, State: Initialized, Role: FOLLOWER
I20260805 23:07:43.867956 22476 consensus_queue.cc:260] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.869105 22478 tablet_bootstrap.cc:492] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Bootstrap starting.
I20260805 23:07:43.869146 22476 ts_tablet_manager.cc:1428] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260805 23:07:43.869570 22023 heartbeater.cc:503] Master 127.3.12.126:37683 was elected leader, sending a full tablet report...
I20260805 23:07:43.875883 22478 tablet_bootstrap.cc:654] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:43.876948 22478 log.cc:824] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:43.877820 22481 tablet_bootstrap.cc:492] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Bootstrap starting.
I20260805 23:07:43.880297 22478 tablet_bootstrap.cc:492] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: No bootstrap required, opened a new log
I20260805 23:07:43.880407 22481 tablet_bootstrap.cc:654] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:43.880532 22478 ts_tablet_manager.cc:1397] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260805 23:07:43.881157 22481 log.cc:824] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:43.883929 22481 tablet_bootstrap.cc:492] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: No bootstrap required, opened a new log
I20260805 23:07:43.884191 22481 ts_tablet_manager.cc:1397] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260805 23:07:43.884761 22478 raft_consensus.cc:348] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.885069 22478 raft_consensus.cc:374] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:43.885139 22478 raft_consensus.cc:729] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 008496c81e3d47719fab994ff681d069, State: Initialized, Role: FOLLOWER
I20260805 23:07:43.885697 22478 consensus_queue.cc:260] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.886826 22478 ts_tablet_manager.cc:1428] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 23:07:43.887228 22424 heartbeater.cc:503] Master 127.3.12.126:37683 was elected leader, sending a full tablet report...
I20260805 23:07:43.889446 22481 raft_consensus.cc:348] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.889719 22481 raft_consensus.cc:374] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:43.889772 22481 raft_consensus.cc:729] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ce7ca484e14aaaa3dfc2be2009da66, State: Initialized, Role: FOLLOWER
I20260805 23:07:43.890239 22481 consensus_queue.cc:260] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.891310 22223 heartbeater.cc:503] Master 127.3.12.126:37683 was elected leader, sending a full tablet report...
I20260805 23:07:43.891342 22481 ts_tablet_manager.cc:1428] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
Aug 05 23:07:43 dist-test-slave-2d19 krb5kdc[20283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971263, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for HTTP/127.3.12.84@KRBTEST.COM
I20260805 23:07:43.933724 22488 raft_consensus.cc:482] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:07:43.933974 22488 raft_consensus.cc:504] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.935479 22488 leader_election.cc:288] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bae81f907b84a209adb07aa4d14bac2 (127.3.12.65:44997), 39ce7ca484e14aaaa3dfc2be2009da66 (127.3.12.66:41847)
W20260805 23:07:43.948204 22024 tablet.cc:2369] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:07:43.951778 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.936761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:38147 (local address 127.3.12.66:41847)
0805 23:07:43.936886 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:43.936890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:43.947267 (+ 10377us) server_negotiation.cc:407] Connection header received
0805 23:07:43.947493 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:43.947497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:43.947554 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:43.947607 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:43.948045 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.948766 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.950410 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.951025 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.951108 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.951179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:43.951549 (+ 370us) server_negotiation.cc:299] Negotiation successful
0805 23:07:43.951636 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:07:43.952370 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.936975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:41847 (local address 127.3.12.67:38147)
0805 23:07:43.947164 (+ 10189us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:43.947181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:43.947346 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:43.947749 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:43.947752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:43.947770 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:43.947906 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.947912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.948917 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.948920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.950258 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.950269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.951282 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.951286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.951392 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.951415 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:07:43.952039 (+ 624us) client_negotiation.cc:240] Negotiation successful
0805 23:07:43.952126 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10101,"thread_start_us":101,"threads_started":1}
I20260805 23:07:43.953097 22144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c73267a284720a32430fcd0d5bd73" candidate_uuid: "008496c81e3d47719fab994ff681d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" is_pre_election: true
I20260805 23:07:43.953464 22144 raft_consensus.cc:2454] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 008496c81e3d47719fab994ff681d069 in term 0.
I20260805 23:07:43.954066 22254 leader_election.cc:302] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 008496c81e3d47719fab994ff681d069, 39ce7ca484e14aaaa3dfc2be2009da66; no voters:
I20260805 23:07:43.954406 22488 raft_consensus.cc:2781] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:07:43.954507 22488 raft_consensus.cc:482] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:07:43.954555 22488 raft_consensus.cc:3037] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:43.955192 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.936602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:47441 (local address 127.3.12.65:44997)
0805 23:07:43.936748 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:43.936753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:43.936767 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:07:43.936823 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:43.936825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:43.936877 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:43.936956 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:43.944816 (+ 7860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.945675 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.953425 (+ 7750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.954094 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.954191 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.954258 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:43.954981 (+ 723us) server_negotiation.cc:299] Negotiation successful
0805 23:07:43.955068 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260805 23:07:43.955790 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.935912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:44997 (local address 127.3.12.67:47441)
0805 23:07:43.936498 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:43.936522 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:43.936636 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:43.944405 (+ 7769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:43.944409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:43.944447 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:43.944625 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.944633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.946387 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.946390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.953138 (+ 6748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:43.953151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.954484 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.954488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:43.954802 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.954829 (+ 27us) client_negotiation.cc:769] Sending connection context
0805 23:07:43.955527 (+ 698us) client_negotiation.cc:240] Negotiation successful
0805 23:07:43.955612 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":139,"threads_started":1}
I20260805 23:07:43.955992 22488 raft_consensus.cc:504] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.956432 22488 leader_election.cc:288] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [CANDIDATE]: Term 1 election: Requested vote from peers 9bae81f907b84a209adb07aa4d14bac2 (127.3.12.65:44997), 39ce7ca484e14aaaa3dfc2be2009da66 (127.3.12.66:41847)
I20260805 23:07:43.956671 21956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c73267a284720a32430fcd0d5bd73" candidate_uuid: "008496c81e3d47719fab994ff681d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bae81f907b84a209adb07aa4d14bac2" is_pre_election: true
I20260805 23:07:43.956954 22144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c73267a284720a32430fcd0d5bd73" candidate_uuid: "008496c81e3d47719fab994ff681d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39ce7ca484e14aaaa3dfc2be2009da66"
I20260805 23:07:43.956957 21956 raft_consensus.cc:2454] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 008496c81e3d47719fab994ff681d069 in term 0.
I20260805 23:07:43.957072 22144 raft_consensus.cc:3037] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:43.957827 21956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568c73267a284720a32430fcd0d5bd73" candidate_uuid: "008496c81e3d47719fab994ff681d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bae81f907b84a209adb07aa4d14bac2"
I20260805 23:07:43.957943 21956 raft_consensus.cc:3037] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:43.958354 22144 raft_consensus.cc:2454] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 008496c81e3d47719fab994ff681d069 in term 1.
I20260805 23:07:43.959252 21956 raft_consensus.cc:2454] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 008496c81e3d47719fab994ff681d069 in term 1.
I20260805 23:07:43.959678 22254 leader_election.cc:302] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 008496c81e3d47719fab994ff681d069, 9bae81f907b84a209adb07aa4d14bac2; no voters:
I20260805 23:07:43.960047 22488 raft_consensus.cc:2781] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:07:43.960296 22488 raft_consensus.cc:686] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 1 LEADER]: Becoming Leader. State: Replica: 008496c81e3d47719fab994ff681d069, State: Running, Role: LEADER
I20260805 23:07:43.960647 22488 consensus_queue.cc:237] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } }
I20260805 23:07:43.964334 21741 catalog_manager.cc:5495] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 reported cstate change: term changed from 0 to 1, leader changed from <none> to 008496c81e3d47719fab994ff681d069 (127.3.12.67). New cstate: current_term: 1 leader_uuid: "008496c81e3d47719fab994ff681d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "008496c81e3d47719fab994ff681d069" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 } health_report { overall_health: UNKNOWN } } }
W20260805 23:07:43.983781 22224 tablet.cc:2369] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:07:43.987112 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:43.982080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.3.12.67:45439)
0805 23:07:43.982213 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:43.982218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:43.982235 (+ 17us) server_negotiation.cc:407] Connection header received
0805 23:07:43.982390 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:43.982393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:43.982470 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:43.982591 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:07:43.983018 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.983762 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.985061 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:43.985381 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:43.985480 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:43.985905 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:07:43.986478 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:07:43.986606 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:43.986785 (+ 179us) server_negotiation.cc:299] Negotiation successful
0805 23:07:43.986858 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:07:43.996718 22144 raft_consensus.cc:1261] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 1 FOLLOWER]: Refusing update from remote peer 008496c81e3d47719fab994ff681d069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 23:07:43.997565 22425 tablet.cc:2369] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:07:43.997658 22494 consensus_queue.cc:1035] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39ce7ca484e14aaaa3dfc2be2009da66" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 41847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:07:44.005513 21956 raft_consensus.cc:1261] T 568c73267a284720a32430fcd0d5bd73 P 9bae81f907b84a209adb07aa4d14bac2 [term 1 FOLLOWER]: Refusing update from remote peer 008496c81e3d47719fab994ff681d069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:07:44.006359 22494 consensus_queue.cc:1035] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bae81f907b84a209adb07aa4d14bac2" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 44997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:07:44.020323 22499 mvcc.cc:205] Tried to move back new op lower bound from 7315338297321328640 to 7315338297186304000. Current Snapshot: MvccSnapshot[applied={T|T < 7315338297321328640}]
I20260805 23:07:44.031117 22502 mvcc.cc:205] Tried to move back new op lower bound from 7315338297321328640 to 7315338297186304000. Current Snapshot: MvccSnapshot[applied={T|T < 7315338297321328640}]
I20260805 23:07:44.038352 21738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:07:44.038568 21738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:07:44.041488 21738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7473bdc6cdba44a2bb855e490be9b1cc: Sending DeleteTablet for 3 replicas of tablet 568c73267a284720a32430fcd0d5bd73
I20260805 23:07:44.043222 22300 tablet_service.cc:1514] Processing DeleteTablet for tablet 568c73267a284720a32430fcd0d5bd73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:44 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:57012
I20260805 23:07:44.043812 21933 tablet_service.cc:1514] Processing DeleteTablet for tablet 568c73267a284720a32430fcd0d5bd73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:44 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:49978
I20260805 23:07:44.044358 22101 tablet_service.cc:1514] Processing DeleteTablet for tablet 568c73267a284720a32430fcd0d5bd73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:44 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:56166
I20260805 23:07:44.045531 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 21797
I20260805 23:07:44.065855 22514 tablet_replica.cc:330] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: stopping tablet replica
I20260805 23:07:44.066287 22514 raft_consensus.cc:2229] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:07:44.066818 22514 raft_consensus.cc:2258] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:07:44.072530 22516 tablet_replica.cc:330] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: stopping tablet replica
I20260805 23:07:44.072926 22516 raft_consensus.cc:2229] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:07:44.073343 22516 raft_consensus.cc:2258] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:07:44.078104 22516 ts_tablet_manager.cc:1905] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:07:44.085528 22514 ts_tablet_manager.cc:1905] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:07:44.089445 22516 ts_tablet_manager.cc:1918] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:07:44.091749 22514 ts_tablet_manager.cc:1918] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:07:44.091888 22514 log.cc:1196] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/568c73267a284720a32430fcd0d5bd73
I20260805 23:07:44.092325 22514 ts_tablet_manager.cc:1939] T 568c73267a284720a32430fcd0d5bd73 P 008496c81e3d47719fab994ff681d069: Deleting consensus metadata
I20260805 23:07:44.093266 22516 log.cc:1196] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/568c73267a284720a32430fcd0d5bd73
I20260805 23:07:44.093663 22516 ts_tablet_manager.cc:1939] T 568c73267a284720a32430fcd0d5bd73 P 39ce7ca484e14aaaa3dfc2be2009da66: Deleting consensus metadata
I20260805 23:07:44.095209 21717 catalog_manager.cc:4841] TS 39ce7ca484e14aaaa3dfc2be2009da66 (127.3.12.66:41847): tablet 568c73267a284720a32430fcd0d5bd73 (table test-table [id=d48045ae118c4118807190acb677c6b0]) successfully deleted
W20260805 23:07:44.095736 21717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:44997 (error 108)
I20260805 23:07:44.097074 21716 catalog_manager.cc:4841] TS 008496c81e3d47719fab994ff681d069 (127.3.12.67:45439): tablet 568c73267a284720a32430fcd0d5bd73 (table test-table [id=d48045ae118c4118807190acb677c6b0]) successfully deleted
I20260805 23:07:44.104450 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 22030
I20260805 23:07:44.151844 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:44.151442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:44997 (local address 127.0.0.1:49980)
0805 23:07:44.151627 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:44.151732 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260805 23:07:44.152124 21717 catalog_manager.cc:4584] TS 9bae81f907b84a209adb07aa4d14bac2 (127.3.12.65:44997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 568c73267a284720a32430fcd0d5bd73: Network error: Client connection negotiation failed: client connection to 127.3.12.65:44997: connect: Connection refused (error 111)
I20260805 23:07:44.157445 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 22227
I20260805 23:07:44.205771 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:44.205347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:44997 (local address 127.0.0.1:49986)
0805 23:07:44.205591 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:44.205661 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260805 23:07:44.206681 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:44.206409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:44997 (local address 127.0.0.1:49996)
0805 23:07:44.206555 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:44.206607 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:44997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260805 23:07:44.212183 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 21694
I20260805 23:07:44.225220 3121 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260805 23:07:44.570967 3121 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260805 23:07:44.571081 3121 mini_ranger.cc:66] Stopping Ranger...
I20260805 23:07:44.926369 3121 mini_ranger.cc:68] Stopped Ranger
2026-08-05 23:07:44.926 UTC [20332] LOG: received smart shutdown request
2026-08-05 23:07:44.931 UTC [20332] LOG: background worker "logical replication launcher" (PID 20342) exited with exit code 1
2026-08-05 23:07:44.931 UTC [20336] LOG: shutting down
2026-08-05 23:07:45.106 UTC [20332] LOG: database system is shut down
2026-08-05T23:07:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50389 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:07:45 dist-test-slave-2d19 krb5kdc[22527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:07:45 dist-test-slave-2d19 krb5kdc[22527](info): set up 2 sockets
Aug 05 23:07:45 dist-test-slave-2d19 krb5kdc[22527](info): commencing operation
krb5kdc: starting...
W20260805 23:07:46.462679 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:07:46 dist-test-slave-2d19 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:07:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:07:46.596040 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41017
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:41017,127.3.12.125:39561,127.3.12.124:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:46.711220 22546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:46.711506 22546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:46.711548 22546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:46.715497 22546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:46.715566 22546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:46.715586 22546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:46.715603 22546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:46.715618 22546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:46.720817 22546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:41017,127.3.12.125:39561,127.3.12.124:42981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:46.721818 22546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:46.722805 22546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:46.732481 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:46.732481 22552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:46.732497 22557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:46.732854 22546 server_base.cc:1034] running on GCE node
I20260805 23:07:46.733421 22546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:46.734565 22546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:46.735757 22546 hybrid_clock.cc:648] HybridClock initialized: now 1785971266735743 us; error 27 us; skew 500 ppm
Aug 05 23:07:46 dist-test-slave-2d19 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:46.738731 22546 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:07:46.739706 22546 webserver.cc:460] Webserver started at http://127.3.12.126:43077/ using document root <none> and password file <none>
I20260805 23:07:46.740386 22546 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:46.740459 22546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:46.740685 22546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:07:46.742841 22546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "766d57228e7c4bdda52a0de8a4556d83"
format_stamp: "Formatted at 2026-08-05 23:07:46 on dist-test-slave-2d19"
server_key: "b61f0c770f43ffaf94157c5d1ab10f10"
server_key_iv: "118daa0f9eb9706891be98836c2a2342"
server_key_version: "encryptionkey@0"
I20260805 23:07:46.743474 22546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "766d57228e7c4bdda52a0de8a4556d83"
format_stamp: "Formatted at 2026-08-05 23:07:46 on dist-test-slave-2d19"
server_key: "b61f0c770f43ffaf94157c5d1ab10f10"
server_key_iv: "118daa0f9eb9706891be98836c2a2342"
server_key_version: "encryptionkey@0"
I20260805 23:07:46.747413 22546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:07:46.750080 22568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:46.751449 22546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 23:07:46.751622 22546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "766d57228e7c4bdda52a0de8a4556d83"
format_stamp: "Formatted at 2026-08-05 23:07:46 on dist-test-slave-2d19"
server_key: "b61f0c770f43ffaf94157c5d1ab10f10"
server_key_iv: "118daa0f9eb9706891be98836c2a2342"
server_key_version: "encryptionkey@0"
I20260805 23:07:46.751744 22546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:46.790372 22546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:46.793756 22546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:46.793972 22546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:46.801658 22546 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:41017
I20260805 23:07:46.801680 22647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:41017 every 8 connection(s)
I20260805 23:07:46.802731 22546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:07:46.803715 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 22546
I20260805 23:07:46.803865 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:07:46.804236 3121 external_mini_cluster.cc:1428] Setting key 9c35265d2569d585be3f5677309b253a
I20260805 23:07:46.806360 22648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:46.810417 22648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.125@KRBTEST.COM" created.
Aug 05 23:07:46 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:46.826749 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:46.812765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41017 (local address 127.0.0.1:58698)
0805 23:07:46.813341 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:46.813385 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:46.814209 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:46.814904 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:46.814908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:46.815230 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:46.815495 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:46.815508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:46.817138 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:46.817141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:46.817881 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:46.817890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:46.818584 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:46.818587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:46.818683 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:46.819352 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:46.819385 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:46.821190 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:46.823654 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:46.823663 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:46.823671 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:46.823908 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:46.824355 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:46.824358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:46.824359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:46.824496 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:46.825322 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:46.825328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:46.825454 (+ 126us) client_negotiation.cc:769] Sending connection context
0805 23:07:46.825604 (+ 150us) client_negotiation.cc:240] Negotiation successful
0805 23:07:46.825879 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":2,"thread_start_us":135,"threads_started":1}
I20260805 23:07:46.826756 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:46.813583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.3.12.126:41017)
0805 23:07:46.813914 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:46.813929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:46.813976 (+ 47us) server_negotiation.cc:407] Connection header received
0805 23:07:46.814531 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:46.814536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:46.814715 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:46.814876 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:46.815645 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:46.816988 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:46.818013 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:46.818418 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:46.818597 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:46.821354 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:46.821363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:46.821378 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:46.821412 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:46.823491 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:46.824037 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:46.824045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:46.824047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:46.824199 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:46.824614 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:46.824620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:46.824625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:46.825192 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:46.825339 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:46.825726 (+ 387us) server_negotiation.cc:299] Negotiation successful
0805 23:07:46.825964 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
I20260805 23:07:46.829524 22648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:07:46.830390 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:46.830092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33832)
0805 23:07:46.830248 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:46.830328 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260805 23:07:46.830622 22578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
I20260805 23:07:46.831533 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:46.831321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33834)
0805 23:07:46.831425 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:46.831476 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260805 23:07:46.831660 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:39561: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab.
Entry for principal kudu/127.3.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab.
I20260805 23:07:46.836014 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.125:39561
--webserver_interface=127.3.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab
--principal=kudu/127.3.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:41017,127.3.12.125:39561,127.3.12.124:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:07:46.880935 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } attempt: 1
I20260805 23:07:46.881851 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:46.881532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33844)
0805 23:07:46.881692 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:46.881769 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260805 23:07:46.882647 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:46.882412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33854)
0805 23:07:46.882544 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:46.882593 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260805 23:07:46.882779 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:39561: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
I20260805 23:07:46.950966 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } attempt: 2
W20260805 23:07:46.951066 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:46.951493 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:46.951557 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260805 23:07:46.951931 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:46.951612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33864)
0805 23:07:46.951768 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:46.951838 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260805 23:07:46.952813 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:46.952577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33876)
0805 23:07:46.952693 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:46.952752 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260805 23:07:46.952946 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:39561: Network error: Client connection negotiation failed: client connection to 127.3.12.125:39561: connect: Connection refused (error 111)
W20260805 23:07:46.956211 22657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:46.956301 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:46.956353 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:46.956394 22657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:46.956447 22657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:46.961861 22657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:41017,127.3.12.125:39561,127.3.12.124:42981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab
--principal=kudu/127.3.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.125:39561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:46.963042 22657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:46.964066 22657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:46.972828 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:46.972827 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:46.972827 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:46.972977 22657 server_base.cc:1034] running on GCE node
I20260805 23:07:46.973830 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:46.974949 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:46.976171 22657 hybrid_clock.cc:648] HybridClock initialized: now 1785971266976140 us; error 52 us; skew 500 ppm
Aug 05 23:07:46 dist-test-slave-2d19 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:46.979334 22657 init.cc:377] Logged in from keytab as kudu/127.3.12.125@KRBTEST.COM (short username kudu)
I20260805 23:07:46.980332 22657 webserver.cc:460] Webserver started at http://127.3.12.125:34019/ using document root <none> and password file <none>
I20260805 23:07:46.980880 22657 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:46.980950 22657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:46.981122 22657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:07:46.983230 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/data/instance:
uuid: "398177bdb18a46bca0cd615b089931dd"
format_stamp: "Formatted at 2026-08-05 23:07:46 on dist-test-slave-2d19"
server_key: "0dcde8db5a66e348086fbfb515352ba6"
server_key_iv: "131e817719553a53bce4e83e7a14e781"
server_key_version: "encryptionkey@0"
I20260805 23:07:46.983762 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/wal/instance:
uuid: "398177bdb18a46bca0cd615b089931dd"
format_stamp: "Formatted at 2026-08-05 23:07:46 on dist-test-slave-2d19"
server_key: "0dcde8db5a66e348086fbfb515352ba6"
server_key_iv: "131e817719553a53bce4e83e7a14e781"
server_key_version: "encryptionkey@0"
I20260805 23:07:46.987630 22657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:07:46.990352 22677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:46.991780 22657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:07:46.991888 22657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/wal
uuid: "398177bdb18a46bca0cd615b089931dd"
format_stamp: "Formatted at 2026-08-05 23:07:46 on dist-test-slave-2d19"
server_key: "0dcde8db5a66e348086fbfb515352ba6"
server_key_iv: "131e817719553a53bce4e83e7a14e781"
server_key_version: "encryptionkey@0"
I20260805 23:07:46.992038 22657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:47.002928 22657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:47.006618 22657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:47.006770 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:47.014078 22657 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.125:39561
I20260805 23:07:47.014145 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.125:39561 every 8 connection(s)
I20260805 23:07:47.015100 22657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
I20260805 23:07:47.017992 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:47.022022 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:07:47.023787 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 22657
I20260805 23:07:47.023954 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-1/wal/instance
I20260805 23:07:47.024312 3121 external_mini_cluster.cc:1428] Setting key 27e7c2f1704cc9622245959f3f1f018c
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.124@KRBTEST.COM" created.
I20260805 23:07:47.037439 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.024161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.3.12.126:41017)
0805 23:07:47.024354 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.024359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.025330 (+ 971us) server_negotiation.cc:407] Connection header received
0805 23:07:47.026298 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.026302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.026367 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.026478 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.027718 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.028308 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.029509 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.029767 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.029879 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.032567 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.032571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.032574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.032611 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.034889 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.035665 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.035669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.035671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.035773 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.036148 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.036152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.036154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.036423 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.036532 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.037118 (+ 586us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.037256 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260805 23:07:47.038455 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.024612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41017 (local address 127.0.0.1:58706)
0805 23:07:47.025127 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.025182 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:47.026072 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:47.026665 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:47.026672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:47.027047 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:47.027550 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.027564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.028457 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.028461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.029355 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.029365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.029937 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.029941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.030115 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.030775 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:47.030792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:47.032379 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:47.035127 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.035136 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.035197 (+ 61us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.035543 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.035878 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.035882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.035884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.036044 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.036532 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:47.036541 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:47.036845 (+ 304us) client_negotiation.cc:769] Sending connection context
0805 23:07:47.037052 (+ 207us) client_negotiation.cc:240] Negotiation successful
0805 23:07:47.037408 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":143,"threads_started":1}
I20260805 23:07:47.040158 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:07:47.044114 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } attempt: 3
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
I20260805 23:07:47.050467 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.040803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33880)
0805 23:07:47.040915 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.040931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:47.041044 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:47.041660 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:47.041663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:47.041854 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:47.041938 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.041943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.042920 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.042926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.043456 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.043463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.044069 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.044072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.044163 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.044794 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:47.044809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:47.046383 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:47.048666 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.048671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.048673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.048921 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.049265 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.049279 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.049284 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.049398 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.049906 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:47.049908 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:47.050013 (+ 105us) client_negotiation.cc:769] Sending connection context
0805 23:07:47.050129 (+ 116us) client_negotiation.cc:240] Negotiation successful
0805 23:07:47.050274 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260805 23:07:47.050697 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.040933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.3.12.125:39561)
0805 23:07:47.041311 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.041319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.041337 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:07:47.041369 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.041373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.041538 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.041684 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.042071 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.042806 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.043580 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.043947 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.044074 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.046546 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.046556 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.046569 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.046589 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.048525 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.049056 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.049063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.049064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.049148 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.049502 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.049506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.049508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.049782 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.049945 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.050244 (+ 299us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.050426 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":185,"threads_started":1}
I20260805 23:07:47.053146 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:07:47.053973 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.053667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40100)
0805 23:07:47.053806 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.053893 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260805 23:07:47.054231 22679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
I20260805 23:07:47.054725 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.044818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.3.12.125:39561)
0805 23:07:47.045163 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.045172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.045185 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:07:47.045233 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.045237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.045291 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.045454 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.045871 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.046491 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.047349 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.048182 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.048320 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.050492 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.050496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.050501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.050591 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.052613 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.053226 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.053229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.053231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.053323 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.053722 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.053725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.053726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.053951 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.054073 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.054398 (+ 325us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.054543 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":107,"threads_started":1}
I20260805 23:07:47.054768 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.044596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33896)
0805 23:07:47.044871 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.044891 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:47.044983 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:47.045475 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:47.045490 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:47.045653 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:47.045753 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.045759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.046644 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.046647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.047229 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.047237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.048320 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.048323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.048436 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.048863 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:47.048882 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:47.050344 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:47.052754 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.052757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.052758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.053139 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.053459 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.053465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.053467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.053579 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.054090 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:47.054094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:47.054212 (+ 118us) client_negotiation.cc:769] Sending connection context
0805 23:07:47.054342 (+ 130us) client_negotiation.cc:240] Negotiation successful
0805 23:07:47.054514 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab.
Entry for principal kudu/127.3.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab.
I20260805 23:07:47.055682 22648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:07:47.055943 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.055713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40110)
0805 23:07:47.055812 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.055868 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260805 23:07:47.056165 22747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:42981: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
I20260805 23:07:47.056535 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.124:42981
--webserver_interface=127.3.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab
--principal=kudu/127.3.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:41017,127.3.12.125:39561,127.3.12.124:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:07:47.056667 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.056166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40112)
0805 23:07:47.056485 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.056538 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
I20260805 23:07:47.057453 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.057193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40114)
0805 23:07:47.057340 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.057397 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260805 23:07:47.057621 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:42981: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
I20260805 23:07:47.088794 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } attempt: 1
I20260805 23:07:47.089737 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.089341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40126)
0805 23:07:47.089546 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.089633 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260805 23:07:47.090598 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.090311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40140)
0805 23:07:47.090431 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.090517 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 23:07:47.090750 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:42981: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
I20260805 23:07:47.105470 22747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } attempt: 1
I20260805 23:07:47.106288 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.105978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40148)
0805 23:07:47.106156 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.106211 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260805 23:07:47.107131 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.106913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40150)
0805 23:07:47.107019 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.107072 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 23:07:47.107303 22747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:42981: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
I20260805 23:07:47.165922 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } attempt: 2
I20260805 23:07:47.166776 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.166447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40160)
0805 23:07:47.166630 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.166696 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260805 23:07:47.167557 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.167337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40170)
0805 23:07:47.167441 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.167486 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 23:07:47.167714 22648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:42981: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
W20260805 23:07:47.172828 22758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:47.173156 22758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:47.173255 22758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260805 23:07:47.175468 22747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } attempt: 2
I20260805 23:07:47.176306 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.175955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40178)
0805 23:07:47.176121 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.176180 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260805 23:07:47.177003 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.176802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40182)
0805 23:07:47.176908 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.176953 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 23:07:47.177172 22747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:42981: Network error: Client connection negotiation failed: client connection to 127.3.12.124:42981: connect: Connection refused (error 111)
W20260805 23:07:47.177909 22758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:47.177989 22758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:47.178035 22758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:47.178072 22758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:47.178108 22758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:47.183629 22758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:41017,127.3.12.125:39561,127.3.12.124:42981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab
--principal=kudu/127.3.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.124:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:47.184826 22758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:47.185877 22758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:47.194481 22766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:47.194487 22765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:47.194538 22769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:47.194769 22758 server_base.cc:1034] running on GCE node
I20260805 23:07:47.196879 22758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:47.197921 22758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:47.199124 22758 hybrid_clock.cc:648] HybridClock initialized: now 1785971267199108 us; error 37 us; skew 500 ppm
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971267, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:47.202015 22758 init.cc:377] Logged in from keytab as kudu/127.3.12.124@KRBTEST.COM (short username kudu)
I20260805 23:07:47.202842 22758 webserver.cc:460] Webserver started at http://127.3.12.124:45479/ using document root <none> and password file <none>
I20260805 23:07:47.203364 22758 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:47.203425 22758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:47.203660 22758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:07:47.205772 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/data/instance:
uuid: "a7d3f60b188a4533a2755e6d6afbf28b"
format_stamp: "Formatted at 2026-08-05 23:07:47 on dist-test-slave-2d19"
server_key: "c1e4539bac7791ef792009ec2403c74b"
server_key_iv: "cf17e4a9bca205c7001aba84337f4674"
server_key_version: "encryptionkey@0"
I20260805 23:07:47.206468 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/wal/instance:
uuid: "a7d3f60b188a4533a2755e6d6afbf28b"
format_stamp: "Formatted at 2026-08-05 23:07:47 on dist-test-slave-2d19"
server_key: "c1e4539bac7791ef792009ec2403c74b"
server_key_iv: "cf17e4a9bca205c7001aba84337f4674"
server_key_version: "encryptionkey@0"
I20260805 23:07:47.210415 22758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:07:47.213006 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:47.214429 22758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:07:47.214573 22758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/wal
uuid: "a7d3f60b188a4533a2755e6d6afbf28b"
format_stamp: "Formatted at 2026-08-05 23:07:47 on dist-test-slave-2d19"
server_key: "c1e4539bac7791ef792009ec2403c74b"
server_key_iv: "cf17e4a9bca205c7001aba84337f4674"
server_key_version: "encryptionkey@0"
I20260805 23:07:47.214694 22758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:47.229326 22758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:47.234035 22758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:47.234282 22758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:47.242659 22758 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.124:42981
I20260805 23:07:47.242677 22852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.124:42981 every 8 connection(s)
I20260805 23:07:47.243681 22758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
I20260805 23:07:47.244453 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 22758
I20260805 23:07:47.244608 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785971095475883-3121-0/minicluster-data/master-2/wal/instance
I20260805 23:07:47.244923 3121 external_mini_cluster.cc:1428] Setting key ebce79b1865dbbc5530a23c60e29ed61
I20260805 23:07:47.247208 22853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:47.251333 22853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971267, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:47.260411 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.246517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.3.12.124:42981)
0805 23:07:47.247008 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.247025 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.247063 (+ 38us) server_negotiation.cc:407] Connection header received
0805 23:07:47.247776 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.247795 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.248133 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.248540 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.249379 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.250372 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.251255 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.251782 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.251965 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.254977 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.254982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.254991 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.255018 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.257128 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.257580 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.257585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.257589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.257756 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.258104 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.258108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.258109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.258605 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.258842 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.259146 (+ 304us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.259423 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":151,"threads_started":1}
I20260805 23:07:47.263726 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.253215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.3.12.126:41017)
0805 23:07:47.253371 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.253375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.253643 (+ 268us) server_negotiation.cc:407] Connection header received
0805 23:07:47.253772 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.253776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.253838 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.253927 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.254658 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.255361 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.256700 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.257004 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.257092 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.259486 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.259489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.259491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.259515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.261121 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.262241 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.262245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.262247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.262347 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.262696 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.262702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.262706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.262972 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.263080 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.263443 (+ 363us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.263572 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260805 23:07:47.263741 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.253135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:41017 (local address 127.0.0.1:58710)
0805 23:07:47.253493 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.253544 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:47.253668 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:47.253992 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:47.253999 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:47.254334 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:47.254525 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.254532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.255522 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.255526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.256564 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.256577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.257124 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.257128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.257260 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.257762 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:47.257777 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:47.259325 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:47.261228 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.261235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.261239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.261448 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.262441 (+ 993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.262445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.262447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.262571 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.263097 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:47.263103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:47.263214 (+ 111us) client_negotiation.cc:769] Sending connection context
0805 23:07:47.263361 (+ 147us) client_negotiation.cc:240] Negotiation successful
0805 23:07:47.263568 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20260805 23:07:47.264948 22853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:07:47.266904 22648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } attempt: 3
I20260805 23:07:47.268321 22747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } attempt: 3
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971267, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
I20260805 23:07:47.274161 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.262942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58724 (local address 127.3.12.126:41017)
0805 23:07:47.263261 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.263284 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.263294 (+ 10us) server_negotiation.cc:407] Connection header received
0805 23:07:47.263319 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.263321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.263353 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.263451 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.264204 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.264882 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.265801 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.266055 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.266169 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.269054 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.269061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.269065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.269098 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.271517 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.272058 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.272064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.272068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.272170 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.272567 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.272573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.272575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.272927 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.273146 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.273810 (+ 664us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.273947 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
I20260805 23:07:47.277490 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.267273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40202)
0805 23:07:47.267422 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.267435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:47.267514 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:47.267742 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:47.267745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:47.267926 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:47.268023 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.268028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.268966 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.268972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.269728 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.269738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.270234 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.270237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.270329 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.270903 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:47.270918 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:47.272365 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:47.274398 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.274401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.274403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.274639 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.275743 (+ 1104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.275748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.275752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.276139 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.276969 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:47.276972 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:47.277082 (+ 110us) client_negotiation.cc:769] Sending connection context
0805 23:07:47.277176 (+ 94us) client_negotiation.cc:240] Negotiation successful
0805 23:07:47.277305 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260805 23:07:47.277882 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.265434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:39561 (local address 127.0.0.1:33898)
0805 23:07:47.265563 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.265577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:47.265651 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:47.265970 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:47.265973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:47.266148 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:47.266233 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.266238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.267036 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.267041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.267841 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.267853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.268679 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.268686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.268801 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.269266 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:47.269284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:47.271499 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:47.275150 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.275153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.275155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.275603 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.276665 (+ 1062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.276668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.276670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.276759 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.277246 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:47.277249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:47.277339 (+ 90us) client_negotiation.cc:769] Sending connection context
0805 23:07:47.277599 (+ 260us) client_negotiation.cc:240] Negotiation successful
0805 23:07:47.277727 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"spinlock_wait_cycles":7936}
I20260805 23:07:47.278133 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.265612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.3.12.125:39561)
0805 23:07:47.265743 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.265747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.265759 (+ 12us) server_negotiation.cc:407] Connection header received
0805 23:07:47.265808 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.265810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.265849 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.265981 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.266362 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.266866 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.267976 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.268414 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.268543 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.271676 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.271683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.271688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.271719 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.275026 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.276405 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.276409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.276414 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.276524 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.276882 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.276888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.276893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.277097 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.277170 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.277526 (+ 356us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.277936 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260805 23:07:47.278717 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.267394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.3.12.124:42981)
0805 23:07:47.267525 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.267529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.267544 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:07:47.267579 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.267581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.267620 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.267715 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.268133 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.268822 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.269851 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.270092 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.270570 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.272512 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.272515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.272518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.272547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.274291 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.275007 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.275013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.275015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.275284 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.276305 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.276309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.276313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.276833 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.278347 (+ 1514us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.278427 (+ 80us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.278561 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":1920}
I20260805 23:07:47.278775 22853 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:07:47.280006 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.268679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40218)
0805 23:07:47.268819 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.268832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:47.268908 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:47.269677 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:47.269683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:47.269858 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:47.269967 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.269976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.271597 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.271601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.272495 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.272504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.273378 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.273382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.273512 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.273911 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:47.273928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:47.275385 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:47.277501 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.277507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.277511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.277797 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.278560 (+ 763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.278564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.278566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.278652 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.279117 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:47.279120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:47.279574 (+ 454us) client_negotiation.cc:769] Sending connection context
0805 23:07:47.279704 (+ 130us) client_negotiation.cc:240] Negotiation successful
0805 23:07:47.279835 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":335}
I20260805 23:07:47.280021 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.269085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.3.12.124:42981)
0805 23:07:47.269434 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.269442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.269463 (+ 21us) server_negotiation.cc:407] Connection header received
0805 23:07:47.269494 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.269497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.269539 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.269623 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.270115 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.271445 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.272825 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.273226 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.274033 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.275524 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.275528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.275533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.275565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.277379 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.278331 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.278334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.278337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.278428 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.278773 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.278778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.278781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.278982 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.279169 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.279772 (+ 603us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.279873 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"spinlock_wait_cycles":4608,"thread_start_us":88,"threads_started":1}
I20260805 23:07:47.282155 22648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83: Bootstrap starting.
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
I20260805 23:07:47.283197 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd: Bootstrap starting.
I20260805 23:07:47.285162 22648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83: Neither blocks nor log segments found. Creating new log.
Aug 05 23:07:47 dist-test-slave-2d19 krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971267, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
I20260805 23:07:47.285921 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:47.286140 22648 log.cc:824] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:47.286808 22747 log.cc:824] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:47.287379 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.275832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.3.12.125:39561)
0805 23:07:47.275944 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.275947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.275958 (+ 11us) server_negotiation.cc:407] Connection header received
0805 23:07:47.275985 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.275987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.276017 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.276114 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.276918 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.277651 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.278690 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.279040 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.279116 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.283703 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.283706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.283709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.283733 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.285333 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.285837 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.285844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.285849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.285942 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.286310 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.286314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.286315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.286514 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.286681 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.287086 (+ 405us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.287219 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260805 23:07:47.288556 22648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83: No bootstrap required, opened a new log
I20260805 23:07:47.288851 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd: No bootstrap required, opened a new log
I20260805 23:07:47.290162 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.279378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:42981 (local address 127.0.0.1:40226)
0805 23:07:47.280348 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:47.280364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:47.280436 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:47.281138 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:47.281141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:47.281296 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:47.281370 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.281376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.282361 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.282364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.283071 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:47.283082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.283807 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.283813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:47.283882 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.284763 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:47.284777 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:47.286349 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:47.288580 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.288583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.288586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.288890 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.289237 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:47.289239 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:47.289241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:47.289314 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:47.289718 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:47.289721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:47.289794 (+ 73us) client_negotiation.cc:769] Sending connection context
0805 23:07:47.289882 (+ 88us) client_negotiation.cc:240] Negotiation successful
0805 23:07:47.289997 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904}
I20260805 23:07:47.290709 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.279435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.3.12.124:42981)
0805 23:07:47.279546 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.279552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.280844 (+ 1292us) server_negotiation.cc:407] Connection header received
0805 23:07:47.280885 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.280887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.280930 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.281017 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.281506 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.282166 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.283244 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.283608 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.283685 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.286487 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.286492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.286494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.286520 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.288463 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.289066 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.289069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.289071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.289149 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.289398 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.289401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.289402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.289593 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.289685 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.290470 (+ 785us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.290579 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:07:47.291778 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:07:47.293195 22648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.293434 22648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:47.293490 22648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766d57228e7c4bdda52a0de8a4556d83, State: Initialized, Role: FOLLOWER
I20260805 23:07:47.293932 22853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b: Bootstrap starting.
I20260805 23:07:47.294005 22648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.293905 22747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.294523 22747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:47.294617 22747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398177bdb18a46bca0cd615b089931dd, State: Initialized, Role: FOLLOWER
I20260805 23:07:47.295069 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.295352 22648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: configured and running, proceeding with master startup.
W20260805 23:07:47.296394 22580 tablet.cc:2369] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:07:47.296754 22747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:07:47.296816 22853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:47.297708 22853 log.cc:824] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:47.299794 22853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b: No bootstrap required, opened a new log
I20260805 23:07:47.303813 22853 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.304066 22853 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:47.304126 22853 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d3f60b188a4533a2755e6d6afbf28b, State: Initialized, Role: FOLLOWER
W20260805 23:07:47.304441 22911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:07:47.304574 22911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:07:47.307224 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.307345 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:07:47.307937 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.308439 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:07:47.305521 22546 master_runner.cc:186] Error getting master registration for 127.3.12.124:42981: OK, instance_id { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" instance_seqno: 1785971267240242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 23:07:47.309312 22546 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 23:07:47.309921 22853 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.311038 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.294809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33916 (local address 127.3.12.125:39561)
0805 23:07:47.294945 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.294949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.296682 (+ 1733us) server_negotiation.cc:407] Connection header received
0805 23:07:47.296993 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.296998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.297045 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.297106 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.300360 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.301062 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.303910 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.304238 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.304341 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.307492 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.307496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.307499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.307529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.309166 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.309658 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.309662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.309664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.309747 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.310060 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.310063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.310065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.310300 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.310415 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.310779 (+ 364us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.310894 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 23:07:47.311347 22853 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:07:47.311443 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.294186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.3.12.126:41017)
0805 23:07:47.294294 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.294298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.296444 (+ 2146us) server_negotiation.cc:407] Connection header received
0805 23:07:47.297248 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.297251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.297293 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.297341 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.299132 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.299902 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.303336 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.303682 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.303762 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.306202 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.306207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.306209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.306234 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.308742 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.309347 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.309352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.309354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.309450 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.310116 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.310120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.310121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.310354 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.310507 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.311171 (+ 664us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.311301 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260805 23:07:47.314637 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.315384 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:07:47.317157 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:47.295528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.3.12.124:42981)
0805 23:07:47.295623 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:47.295627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:47.304563 (+ 8936us) server_negotiation.cc:407] Connection header received
0805 23:07:47.304759 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:47.304763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:47.304818 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:47.304875 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:47.306790 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.307480 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.308639 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:47.308995 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:47.309081 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:47.313065 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:47.313069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:47.313072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:47.313097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:47.314824 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.315349 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.315352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.315354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.315560 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:47.315894 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:47.315897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:47.315899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:47.316126 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:47.316290 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:47.316898 (+ 608us) server_negotiation.cc:299] Negotiation successful
0805 23:07:47.317020 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260805 23:07:47.317543 22928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:07:47.317620 22928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:07:47.311285 22657 master_runner.cc:186] Error getting master registration for 127.3.12.124:42981: OK, instance_id { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" instance_seqno: 1785971267240242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 23:07:47.319818 22657 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 23:07:47.321380 22758 master_runner.cc:263] No leader master found from master a7d3f60b188a4533a2755e6d6afbf28b
I20260805 23:07:47.321442 22758 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260805 23:07:47.324904 22895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:07:47.325105 22895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:07:47.380417 22875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:07:47.380663 22875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.381850 22875 leader_election.cc:288] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 398177bdb18a46bca0cd615b089931dd (127.3.12.125:39561), a7d3f60b188a4533a2755e6d6afbf28b (127.3.12.124:42981)
I20260805 23:07:47.382642 22711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "766d57228e7c4bdda52a0de8a4556d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398177bdb18a46bca0cd615b089931dd" is_pre_election: true
I20260805 23:07:47.383011 22711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 766d57228e7c4bdda52a0de8a4556d83 in term 0.
I20260805 23:07:47.383468 22578 leader_election.cc:302] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398177bdb18a46bca0cd615b089931dd, 766d57228e7c4bdda52a0de8a4556d83; no voters:
I20260805 23:07:47.383793 22875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:07:47.383879 22875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:07:47.383930 22875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:47.385367 22821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "766d57228e7c4bdda52a0de8a4556d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" is_pre_election: true
I20260805 23:07:47.385742 22821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 766d57228e7c4bdda52a0de8a4556d83 in term 0.
I20260805 23:07:47.387454 22875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.387838 22875 leader_election.cc:288] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [CANDIDATE]: Term 1 election: Requested vote from peers 398177bdb18a46bca0cd615b089931dd (127.3.12.125:39561), a7d3f60b188a4533a2755e6d6afbf28b (127.3.12.124:42981)
I20260805 23:07:47.388302 22711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "766d57228e7c4bdda52a0de8a4556d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398177bdb18a46bca0cd615b089931dd"
I20260805 23:07:47.388464 22711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:47.389855 22711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 766d57228e7c4bdda52a0de8a4556d83 in term 1.
I20260805 23:07:47.390200 22578 leader_election.cc:302] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398177bdb18a46bca0cd615b089931dd, 766d57228e7c4bdda52a0de8a4556d83; no voters:
I20260805 23:07:47.390676 22821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "766d57228e7c4bdda52a0de8a4556d83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d3f60b188a4533a2755e6d6afbf28b"
I20260805 23:07:47.390794 22821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:47.392165 22821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 766d57228e7c4bdda52a0de8a4556d83 in term 1.
I20260805 23:07:47.392654 22875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:07:47.392902 22875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [term 1 LEADER]: Becoming Leader. State: Replica: 766d57228e7c4bdda52a0de8a4556d83, State: Running, Role: LEADER
I20260805 23:07:47.393280 22875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } }
I20260805 23:07:47.399106 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 766d57228e7c4bdda52a0de8a4556d83. Latest consensus state: current_term: 1 leader_uuid: "766d57228e7c4bdda52a0de8a4556d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.399240 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:47.399669 22937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:07:47.400709 22937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:07:47.407886 22821 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [term 1 FOLLOWER]: Refusing update from remote peer 766d57228e7c4bdda52a0de8a4556d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:07:47.408618 22932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:07:47.412948 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 766d57228e7c4bdda52a0de8a4556d83. Latest consensus state: current_term: 1 leader_uuid: "766d57228e7c4bdda52a0de8a4556d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.413095 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:07:47.416509 22711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [term 1 FOLLOWER]: Refusing update from remote peer 766d57228e7c4bdda52a0de8a4556d83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:07:47.417097 22875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:07:47.421389 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 766d57228e7c4bdda52a0de8a4556d83. Latest consensus state: current_term: 1 leader_uuid: "766d57228e7c4bdda52a0de8a4556d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.421527 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:07:47.427436 22932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "766d57228e7c4bdda52a0de8a4556d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.427580 22932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:47.431478 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "766d57228e7c4bdda52a0de8a4556d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.431607 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:07:47.432296 22938 mvcc.cc:205] Tried to move back new op lower bound from 7315338311295037440 to 7315338311257726976. Current Snapshot: MvccSnapshot[applied={T|T < 7315338311295037440}]
I20260805 23:07:47.432478 22944 mvcc.cc:205] Tried to move back new op lower bound from 7315338311295037440 to 7315338311257726976. Current Snapshot: MvccSnapshot[applied={T|T < 7315338311295037440}]
I20260805 23:07:47.434017 22937 catalog_manager.cc:1347] Generated new cluster ID: cc7a642986a94afbbb3abf91d03dd98f
I20260805 23:07:47.434091 22937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:07:47.445324 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "766d57228e7c4bdda52a0de8a4556d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.445488 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:07:47.444218 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "766d57228e7c4bdda52a0de8a4556d83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766d57228e7c4bdda52a0de8a4556d83" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41017 } } peers { permanent_uuid: "398177bdb18a46bca0cd615b089931dd" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 39561 } } peers { permanent_uuid: "a7d3f60b188a4533a2755e6d6afbf28b" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 42981 } } }
I20260805 23:07:47.447667 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:47.453317 22940 mvcc.cc:205] Tried to move back new op lower bound from 7315338311295037440 to 7315338311257726976. Current Snapshot: MvccSnapshot[applied={T|T < 7315338311295037440}]
I20260805 23:07:47.475215 22937 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:07:47.476588 22937 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:07:47.515113 22937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 766d57228e7c4bdda52a0de8a4556d83: Generated new TSK 0
I20260805 23:07:47.516104 22937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:07:48.305948 22911 catalog_manager.cc:1259] Loaded cluster ID: cc7a642986a94afbbb3abf91d03dd98f
I20260805 23:07:48.306102 22911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd: loading cluster ID for follower catalog manager: success
I20260805 23:07:48.307731 22911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd: acquiring CA information for follower catalog manager: success
I20260805 23:07:48.308951 22911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 398177bdb18a46bca0cd615b089931dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 23:07:48.318899 22928 catalog_manager.cc:1259] Loaded cluster ID: cc7a642986a94afbbb3abf91d03dd98f
I20260805 23:07:48.318985 22928 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b: loading cluster ID for follower catalog manager: success
I20260805 23:07:48.320631 22928 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b: acquiring CA information for follower catalog manager: success
I20260805 23:07:48.321842 22928 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a7d3f60b188a4533a2755e6d6afbf28b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 23:07:48.587903 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 22546
I20260805 23:07:48.603765 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 22657
I20260805 23:07:48.614818 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 22758
2026-08-05T23:07:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3390 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:07:48 dist-test-slave-2d19 krb5kdc[22960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:07:48 dist-test-slave-2d19 krb5kdc[22960](info): set up 2 sockets
Aug 05 23:07:48 dist-test-slave-2d19 krb5kdc[22960](info): commencing operation
krb5kdc: starting...
W20260805 23:07:49.920564 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:07:50 dist-test-slave-2d19 krb5kdc[22960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:07:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:07:50.055827 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42277
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:50.168504 22983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:50.168797 22983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:50.168869 22983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:50.172894 22983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:50.172972 22983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:50.173009 22983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:50.173070 22983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:50.173105 22983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:50.178182 22983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:42277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:50.179270 22983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:50.180315 22983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:50.189088 22993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:50.189113 22997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:50.189317 22983 server_base.cc:1034] running on GCE node
W20260805 23:07:50.189474 22991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:50.190099 22983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:50.191208 22983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:50.192396 22983 hybrid_clock.cc:648] HybridClock initialized: now 1785971270192377 us; error 32 us; skew 500 ppm
Aug 05 23:07:50 dist-test-slave-2d19 krb5kdc[22960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971270, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:50.195153 22983 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:07:50.196036 22983 webserver.cc:460] Webserver started at http://127.3.12.126:43113/ using document root <none> and password file <none>
I20260805 23:07:50.196625 22983 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:50.196688 22983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:50.196851 22983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:07:50.198841 22983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "aeab426d33604fb6997d227d7e4bd9fd"
format_stamp: "Formatted at 2026-08-05 23:07:50 on dist-test-slave-2d19"
server_key: "f4d91f8abcb7406b9ce7fc1b1d620c58"
server_key_iv: "37815faef70e5ed58f4fd094d439a2d4"
server_key_version: "encryptionkey@0"
I20260805 23:07:50.199371 22983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "aeab426d33604fb6997d227d7e4bd9fd"
format_stamp: "Formatted at 2026-08-05 23:07:50 on dist-test-slave-2d19"
server_key: "f4d91f8abcb7406b9ce7fc1b1d620c58"
server_key_iv: "37815faef70e5ed58f4fd094d439a2d4"
server_key_version: "encryptionkey@0"
I20260805 23:07:50.203182 22983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260805 23:07:50.205637 23006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:50.206892 22983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:07:50.207000 22983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "aeab426d33604fb6997d227d7e4bd9fd"
format_stamp: "Formatted at 2026-08-05 23:07:50 on dist-test-slave-2d19"
server_key: "f4d91f8abcb7406b9ce7fc1b1d620c58"
server_key_iv: "37815faef70e5ed58f4fd094d439a2d4"
server_key_version: "encryptionkey@0"
I20260805 23:07:50.207121 22983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:50.228428 22983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:50.231765 22983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:50.231942 22983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:50.239408 22983 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:42277
I20260805 23:07:50.239450 23088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:42277 every 8 connection(s)
I20260805 23:07:50.240413 22983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:07:50.242944 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 22983
I20260805 23:07:50.243113 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:07:50.243170 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:50.243425 3121 external_mini_cluster.cc:1428] Setting key def335a0969d6a41b6cdd63137482672
I20260805 23:07:50.250541 23089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd: Bootstrap starting.
Aug 05 23:07:50 dist-test-slave-2d19 krb5kdc[22960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:50.253111 23089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:50.254051 23089 log.cc:824] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:50.256488 23089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd: No bootstrap required, opened a new log
I20260805 23:07:50.259220 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:50.244870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.3.12.126:42277)
0805 23:07:50.245310 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:50.245330 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:50.245375 (+ 45us) server_negotiation.cc:407] Connection header received
0805 23:07:50.246092 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:50.246116 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:50.246542 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:50.246938 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:50.247650 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:50.249275 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:50.250167 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:50.250846 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:50.251017 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:50.253641 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:50.253647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:50.253663 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:50.253694 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:50.255922 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:50.256476 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:50.256482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:50.256487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:50.256633 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:50.256928 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:50.256931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:50.256933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:50.257435 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:50.257639 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:50.257843 (+ 204us) server_negotiation.cc:299] Negotiation successful
0805 23:07:50.258150 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260805 23:07:50.260980 23089 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeab426d33604fb6997d227d7e4bd9fd" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42277 } }
I20260805 23:07:50.261281 23089 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:50.261336 23089 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeab426d33604fb6997d227d7e4bd9fd, State: Initialized, Role: FOLLOWER
I20260805 23:07:50.261811 23089 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeab426d33604fb6997d227d7e4bd9fd" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42277 } }
I20260805 23:07:50.261976 23089 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:07:50.262043 23089 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:07:50.262144 23089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:50.262331 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:07:50.263159 23089 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeab426d33604fb6997d227d7e4bd9fd" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42277 } }
I20260805 23:07:50.263518 23089 leader_election.cc:302] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeab426d33604fb6997d227d7e4bd9fd; no voters:
I20260805 23:07:50.263861 23089 leader_election.cc:288] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:07:50.263938 23095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:07:50.264201 23095 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [term 1 LEADER]: Becoming Leader. State: Replica: aeab426d33604fb6997d227d7e4bd9fd, State: Running, Role: LEADER
I20260805 23:07:50.264559 23095 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeab426d33604fb6997d227d7e4bd9fd" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42277 } }
I20260805 23:07:50.265275 23089 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:07:50.266368 23098 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeab426d33604fb6997d227d7e4bd9fd. Latest consensus state: current_term: 1 leader_uuid: "aeab426d33604fb6997d227d7e4bd9fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeab426d33604fb6997d227d7e4bd9fd" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42277 } } }
I20260805 23:07:50.266404 23100 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeab426d33604fb6997d227d7e4bd9fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeab426d33604fb6997d227d7e4bd9fd" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42277 } } }
I20260805 23:07:50.266494 23098 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:50.266513 23100 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:50.266891 23113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:07:50.269759 23113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:07:50.275373 23113 catalog_manager.cc:1347] Generated new cluster ID: 7a0bddb14e5945c6acda62ace2ef7c4c
I20260805 23:07:50.275436 23113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:07:50.302901 23113 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:07:50.304239 23113 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:07:50.317958 23113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeab426d33604fb6997d227d7e4bd9fd: Generated new TSK 0
I20260805 23:07:50.318739 23113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:07:50.369735 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 22983
2026-08-05T23:07:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1755 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:07:50 dist-test-slave-2d19 krb5kdc[23134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:07:50 dist-test-slave-2d19 krb5kdc[23134](info): set up 2 sockets
Aug 05 23:07:50 dist-test-slave-2d19 krb5kdc[23134](info): commencing operation
krb5kdc: starting...
W20260805 23:07:51.590781 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:07:51 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:07:51.724309 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37039
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:37039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:51.838415 23152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:51.838747 23152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:51.838830 23152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:51.842875 23152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:51.842973 23152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:51.843012 23152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:51.843051 23152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:51.843082 23152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:51.848361 23152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:37039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:51.849429 23152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:51.850482 23152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:51.859408 23162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:51.859408 23160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:51.859445 23158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:51.859652 23152 server_base.cc:1034] running on GCE node
I20260805 23:07:51.860363 23152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:51.861497 23152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:51.862710 23152 hybrid_clock.cc:648] HybridClock initialized: now 1785971271862677 us; error 43 us; skew 500 ppm
Aug 05 23:07:51 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971271, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:51.865594 23152 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:07:51.866535 23152 webserver.cc:460] Webserver started at http://127.3.12.126:41609/ using document root <none> and password file <none>
I20260805 23:07:51.867125 23152 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:51.867220 23152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:51.867457 23152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:07:51.869556 23152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "628c3b4d034f45829c11dc189afb0163"
format_stamp: "Formatted at 2026-08-05 23:07:51 on dist-test-slave-2d19"
server_key: "e6e4d0ad104ff34ae4167fe41b914f81"
server_key_iv: "cec4e2cb02c22c4eb9de3407f402d6a5"
server_key_version: "encryptionkey@0"
I20260805 23:07:51.870131 23152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "628c3b4d034f45829c11dc189afb0163"
format_stamp: "Formatted at 2026-08-05 23:07:51 on dist-test-slave-2d19"
server_key: "e6e4d0ad104ff34ae4167fe41b914f81"
server_key_iv: "cec4e2cb02c22c4eb9de3407f402d6a5"
server_key_version: "encryptionkey@0"
I20260805 23:07:51.873919 23152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:07:51.876452 23172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:51.877674 23152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:07:51.877840 23152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "628c3b4d034f45829c11dc189afb0163"
format_stamp: "Formatted at 2026-08-05 23:07:51 on dist-test-slave-2d19"
server_key: "e6e4d0ad104ff34ae4167fe41b914f81"
server_key_iv: "cec4e2cb02c22c4eb9de3407f402d6a5"
server_key_version: "encryptionkey@0"
I20260805 23:07:51.877997 23152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:51.907487 23152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:51.910730 23152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:51.910960 23152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:51.918866 23152 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:37039
I20260805 23:07:51.919173 23242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:37039 every 8 connection(s)
I20260805 23:07:51.919915 23152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:07:51.921340 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 23152
I20260805 23:07:51.921446 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:07:51.921782 3121 external_mini_cluster.cc:1428] Setting key cccefa873a65d960ce3c55ce31bb65ab
I20260805 23:07:51.923206 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:51.930050 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Bootstrap starting.
Aug 05 23:07:51 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:51.932471 23243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:51.933468 23243 log.cc:824] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:51.935437 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: No bootstrap required, opened a new log
I20260805 23:07:51.937198 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:51.923287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.3.12.126:37039)
0805 23:07:51.923761 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:51.923782 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:51.923825 (+ 43us) server_negotiation.cc:407] Connection header received
0805 23:07:51.924708 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:51.924732 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:51.925144 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:51.925513 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:51.926211 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:51.927717 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:51.928669 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:51.929363 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:51.929533 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:51.932049 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:51.932055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:51.932071 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:51.932101 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:51.934158 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:51.934632 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:51.934637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:51.934642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:51.934750 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:51.935048 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:51.935051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:51.935053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:51.935449 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:51.935658 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:51.935948 (+ 290us) server_negotiation.cc:299] Negotiation successful
0805 23:07:51.936233 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
I20260805 23:07:51.938829 23243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:51.939093 23243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:51.939157 23243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628c3b4d034f45829c11dc189afb0163, State: Initialized, Role: FOLLOWER
I20260805 23:07:51.939633 23243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:51.939786 23243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:07:51.939834 23243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:07:51.939904 23243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:51.940824 23243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:51.941109 23243 leader_election.cc:302] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 628c3b4d034f45829c11dc189afb0163; no voters:
I20260805 23:07:51.941365 23243 leader_election.cc:288] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:07:51.941500 23251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:07:51.941744 23251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 1 LEADER]: Becoming Leader. State: Replica: 628c3b4d034f45829c11dc189afb0163, State: Running, Role: LEADER
I20260805 23:07:51.942076 23251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:51.942641 23243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:07:51.943712 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 628c3b4d034f45829c11dc189afb0163. Latest consensus state: current_term: 1 leader_uuid: "628c3b4d034f45829c11dc189afb0163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } } }
I20260805 23:07:51.943864 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:51.944295 23268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:07:51.944563 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "628c3b4d034f45829c11dc189afb0163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } } }
I20260805 23:07:51.944763 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:51.947016 23268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:07:51.952592 23268 catalog_manager.cc:1347] Generated new cluster ID: 4a73ef5de62f4fc4b31cddca8ca60ec4
I20260805 23:07:51.952667 23268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:07:51.990881 23268 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:07:51.992199 23268 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:07:52.002592 23268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Generated new TSK 0
I20260805 23:07:52.003441 23268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:07:52.035813 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37039
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:52.149724 23281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:52.149986 23281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:52.150031 23281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:52.154099 23281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:52.154160 23281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:52.154222 23281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:07:52.159391 23281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:52.160524 23281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:52.161554 23281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:52.171815 23288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:52.171803 23289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:52.172052 23281 server_base.cc:1034] running on GCE node
W20260805 23:07:52.171803 23292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:52.172616 23281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:52.173360 23281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:52.174569 23281 hybrid_clock.cc:648] HybridClock initialized: now 1785971272174547 us; error 32 us; skew 500 ppm
Aug 05 23:07:52 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:52.177662 23281 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:07:52.178613 23281 webserver.cc:460] Webserver started at http://127.3.12.65:46419/ using document root <none> and password file <none>
I20260805 23:07:52.179211 23281 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:52.179307 23281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:52.179562 23281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:07:52.181926 23281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "764ee773f90e4b7a95d3b79d5bda74fb"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2fc060d10a33dde404787ac275f828f4"
server_key_iv: "5cdbb8dcacc37a76ca75734edf758567"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.182614 23281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "764ee773f90e4b7a95d3b79d5bda74fb"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2fc060d10a33dde404787ac275f828f4"
server_key_iv: "5cdbb8dcacc37a76ca75734edf758567"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.186599 23281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:07:52.189417 23302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.190697 23281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:07:52.190851 23281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "764ee773f90e4b7a95d3b79d5bda74fb"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2fc060d10a33dde404787ac275f828f4"
server_key_iv: "5cdbb8dcacc37a76ca75734edf758567"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.190979 23281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:52.211066 23281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:52.214625 23281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:52.214846 23281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:52.215478 23281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:52.216429 23281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:52.216511 23281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.216584 23281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:52.216637 23281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.226997 23281 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:35869
I20260805 23:07:52.227017 23472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:35869 every 8 connection(s)
I20260805 23:07:52.227957 23281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:07:52.233304 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 23281
I20260805 23:07:52.233436 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:07:52.233762 3121 external_mini_cluster.cc:1428] Setting key 05ea4afb2019f7ce2e5250e85fd202de
Aug 05 23:07:52 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:52.244467 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.229834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:47201 (local address 127.3.12.126:37039)
0805 23:07:52.230027 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.230032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.231017 (+ 985us) server_negotiation.cc:407] Connection header received
0805 23:07:52.232409 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.232412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.232461 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.232540 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:52.234131 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.234812 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.236477 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.236736 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.236831 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.240012 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:52.240019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:52.240025 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:52.240060 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:52.242006 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:52.242604 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:52.242609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:52.242613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:52.242716 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:52.243073 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:52.243078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:52.243081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:52.243413 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:52.243565 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.244080 (+ 515us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.244224 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":82}
I20260805 23:07:52.245301 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.230384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:47201)
0805 23:07:52.230853 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.230898 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:52.232105 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:52.232718 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:52.232725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:52.233205 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:52.233954 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.233976 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.234953 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.234957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.236330 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.236346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.236904 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.236911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.237184 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.237961 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:52.237987 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:52.239825 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:52.242129 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:52.242135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:52.242222 (+ 87us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:52.242492 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:52.242823 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:52.242825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:52.242827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:52.242969 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:52.243565 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:52.243570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:52.243833 (+ 263us) client_negotiation.cc:769] Sending connection context
0805 23:07:52.243997 (+ 164us) client_negotiation.cc:240] Negotiation successful
0805 23:07:52.244336 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:07:52.246673 23474 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37039
I20260805 23:07:52.246978 23474 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:52.247536 23474 heartbeater.cc:511] Master 127.3.12.126:37039 requested a full tablet report, sending...
I20260805 23:07:52.249120 23194 ts_manager.cc:194] Registered new tserver with Master: 764ee773f90e4b7a95d3b79d5bda74fb (127.3.12.65:35869)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:07:52.250454 23194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:47201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:07:52.267944 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37039
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:52.382881 23484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:52.383150 23484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:52.383219 23484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:52.387799 23484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:52.387873 23484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:52.388017 23484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:07:52.393554 23484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:52.394699 23484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:52.395763 23484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:52.405236 23494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:52.405229 23491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:52.405285 23492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:52.405498 23484 server_base.cc:1034] running on GCE node
I20260805 23:07:52.406085 23484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:52.406649 23484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:52.407821 23484 hybrid_clock.cc:648] HybridClock initialized: now 1785971272407789 us; error 42 us; skew 500 ppm
Aug 05 23:07:52 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:52.410686 23484 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:07:52.411612 23484 webserver.cc:460] Webserver started at http://127.3.12.66:37899/ using document root <none> and password file <none>
I20260805 23:07:52.412218 23484 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:52.412372 23484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:52.412616 23484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:07:52.414703 23484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "b955aa5503b947af9c2d77c1ade0c5a0"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "9e75a607616bd5a536ca0c2276742b28"
server_key_iv: "1e3ce28fc91bc2867adc16c59756a05a"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.415289 23484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "b955aa5503b947af9c2d77c1ade0c5a0"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "9e75a607616bd5a536ca0c2276742b28"
server_key_iv: "1e3ce28fc91bc2867adc16c59756a05a"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.419165 23484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:07:52.421799 23503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.423158 23484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:07:52.423368 23484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "b955aa5503b947af9c2d77c1ade0c5a0"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "9e75a607616bd5a536ca0c2276742b28"
server_key_iv: "1e3ce28fc91bc2867adc16c59756a05a"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.423522 23484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:52.447479 23484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:52.450618 23484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:52.450832 23484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:52.451464 23484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:52.452397 23484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:52.452466 23484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.452527 23484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:52.452580 23484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.461920 23484 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:45055
I20260805 23:07:52.461979 23669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:45055 every 8 connection(s)
I20260805 23:07:52.462891 23484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:07:52.465826 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 23484
I20260805 23:07:52.465970 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:07:52.466320 3121 external_mini_cluster.cc:1428] Setting key b45f8c2d4b41ff8f1ce026085c5e0102
Aug 05 23:07:52 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:07:52.478860 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.464868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:48397 (local address 127.3.12.126:37039)
0805 23:07:52.465020 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.465023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.466143 (+ 1120us) server_negotiation.cc:407] Connection header received
0805 23:07:52.467730 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.467734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.467791 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.467858 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:52.469195 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.470040 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.471954 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.472201 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.472340 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.474947 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:52.474951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:52.474954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:52.474981 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:52.476558 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:52.477126 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:52.477130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:52.477132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:52.477221 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:52.477648 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:52.477653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:52.477654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:52.477851 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:52.477969 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.478543 (+ 574us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.478683 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:07:52.479773 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.465261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:48397)
0805 23:07:52.465845 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.465977 (+ 132us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:52.467517 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:52.468061 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:52.468068 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:52.468550 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:52.469039 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.469053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.470233 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.470239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.471809 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.471821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.472396 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.472400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.472583 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.473230 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:52.473248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:52.474751 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:52.476687 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:52.476694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:52.476712 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:52.477011 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:52.477327 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:52.477330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:52.477332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:52.477547 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:52.477977 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:52.477988 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:52.478255 (+ 267us) client_negotiation.cc:769] Sending connection context
0805 23:07:52.478472 (+ 217us) client_negotiation.cc:240] Negotiation successful
0805 23:07:52.478857 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
I20260805 23:07:52.481216 23671 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37039
I20260805 23:07:52.481508 23671 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:52.482172 23671 heartbeater.cc:511] Master 127.3.12.126:37039 requested a full tablet report, sending...
I20260805 23:07:52.483445 23194 ts_manager.cc:194] Registered new tserver with Master: b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
I20260805 23:07:52.484314 23194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:48397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:07:52.499027 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37039
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:52.615978 23679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:52.616319 23679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:52.616385 23679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:52.620451 23679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:52.620512 23679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:52.620571 23679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:07:52.625867 23679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:52.626998 23679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:52.628110 23679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:52.638180 23687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:52.638180 23689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:52.638453 23679 server_base.cc:1034] running on GCE node
W20260805 23:07:52.638216 23686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:52.638993 23679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:52.639657 23679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:52.640851 23679 hybrid_clock.cc:648] HybridClock initialized: now 1785971272640820 us; error 41 us; skew 500 ppm
Aug 05 23:07:52 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:52.649034 23679 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:07:52.649948 23679 webserver.cc:460] Webserver started at http://127.3.12.67:35533/ using document root <none> and password file <none>
I20260805 23:07:52.650518 23679 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:52.650581 23679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:52.650763 23679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:07:52.652916 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "fe7a779e84604edb9972cdbc2a5aa932"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2d9bbd0f9a6b5dfa446c82cba1fd7495"
server_key_iv: "5f6c198befceee7276e95eb36c4febe6"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.653458 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "fe7a779e84604edb9972cdbc2a5aa932"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2d9bbd0f9a6b5dfa446c82cba1fd7495"
server_key_iv: "5f6c198befceee7276e95eb36c4febe6"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.657423 23679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 23:07:52.660012 23699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.661509 23679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260805 23:07:52.661693 23679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "fe7a779e84604edb9972cdbc2a5aa932"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2d9bbd0f9a6b5dfa446c82cba1fd7495"
server_key_iv: "5f6c198befceee7276e95eb36c4febe6"
server_key_version: "encryptionkey@0"
I20260805 23:07:52.661823 23679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:52.679929 23679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:52.683167 23679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:52.683374 23679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:52.683972 23679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:52.684943 23679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:52.685034 23679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.685127 23679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:52.685178 23679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:52.694365 23679 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:43739
I20260805 23:07:52.694392 23876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:43739 every 8 connection(s)
I20260805 23:07:52.695365 23679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:07:52.696930 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 23679
I20260805 23:07:52.697088 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:07:52.697443 3121 external_mini_cluster.cc:1428] Setting key 07b19725b04177d06e46a8e18bd75ebf
Aug 05 23:07:52 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971272, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:52.708662 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.697248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:40613 (local address 127.3.12.126:37039)
0805 23:07:52.697431 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.697436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.698328 (+ 892us) server_negotiation.cc:407] Connection header received
0805 23:07:52.699629 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.699632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.699689 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.699767 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:52.701037 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.701571 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.702686 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.702902 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.702993 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.705521 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:52.705524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:52.705526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:52.705550 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:52.706804 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:52.707270 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:52.707273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:52.707274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:52.707352 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:52.707653 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:52.707656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:52.707657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:52.707825 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:52.707902 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.708415 (+ 513us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.708535 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260805 23:07:52.709496 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.697672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:40613)
0805 23:07:52.698162 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.698218 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:52.699452 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:52.699929 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:52.699936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:52.700381 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:52.700896 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.700911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.701689 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.701691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.702573 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.702582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.703035 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.703037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.703185 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.703823 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:52.703840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:52.705350 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:52.706907 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:52.706912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:52.706923 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:52.707177 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:52.707436 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:52.707438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:52.707439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:52.707569 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:52.707915 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:52.707920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:52.708142 (+ 222us) client_negotiation.cc:769] Sending connection context
0805 23:07:52.708374 (+ 232us) client_negotiation.cc:240] Negotiation successful
0805 23:07:52.708628 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":144,"threads_started":1}
I20260805 23:07:52.710939 23877 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37039
I20260805 23:07:52.711225 23877 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:52.711782 23877 heartbeater.cc:511] Master 127.3.12.126:37039 requested a full tablet report, sending...
I20260805 23:07:52.712826 23192 ts_manager.cc:194] Registered new tserver with Master: fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739)
I20260805 23:07:52.713380 23192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:40613
I20260805 23:07:52.723423 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:07:52.733384 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.725519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.3.12.126:37039)
0805 23:07:52.725682 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.725687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.725844 (+ 157us) server_negotiation.cc:407] Connection header received
0805 23:07:52.726067 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.726070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.726121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.726210 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:52.727049 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.727579 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.728590 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.728809 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.728922 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.730339 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:52.730343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:52.730345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:52.730375 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:52.731572 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:52.732032 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:52.732036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:52.732038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:52.732126 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:52.732458 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:52.732462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:52.732464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:52.732684 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:52.732831 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.733083 (+ 252us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.733201 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:07:52.736647 23192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:07:52.739660 23192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:07:52.756155 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.749807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.3.12.66:45055)
0805 23:07:52.750909 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.750921 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.750948 (+ 27us) server_negotiation.cc:407] Connection header received
0805 23:07:52.751033 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.751039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.751284 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.751446 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:52.751759 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.752769 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.754782 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.755576 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.755683 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.755842 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.755890 (+ 48us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.755984 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":128,"threads_started":1}
I20260805 23:07:52.756141 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.750191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.3.12.65:35869)
0805 23:07:52.750963 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.750972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.751000 (+ 28us) server_negotiation.cc:407] Connection header received
0805 23:07:52.751357 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.751362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.751509 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.751666 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:52.752165 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.753241 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.754589 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.755411 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.755534 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.755710 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.755892 (+ 182us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.755984 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":79,"threads_started":1}
I20260805 23:07:52.756217 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.750156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:45055 (local address 127.0.0.1:44882)
0805 23:07:52.750611 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.750626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:52.750713 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:52.751432 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:52.751438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:52.751460 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:52.751618 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.751626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.752904 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.752909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.754645 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.754658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.755697 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.755701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.755773 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.755804 (+ 31us) client_negotiation.cc:769] Sending connection context
0805 23:07:52.755860 (+ 56us) client_negotiation.cc:240] Negotiation successful
0805 23:07:52.756032 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":119,"thread_start_us":84,"threads_started":1}
I20260805 23:07:52.756217 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.749451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:35869 (local address 127.0.0.1:49298)
0805 23:07:52.750543 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.750558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:52.751212 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:52.751889 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:52.751892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:52.751924 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:52.752024 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.752030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.753352 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.753358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.754456 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.754466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.755540 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.755543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.755674 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.755722 (+ 48us) client_negotiation.cc:769] Sending connection context
0805 23:07:52.755843 (+ 121us) client_negotiation.cc:240] Negotiation successful
0805 23:07:52.756032 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":62,"thread_start_us":137,"threads_started":1}
I20260805 23:07:52.758075 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.748669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43739 (local address 127.0.0.1:34264)
0805 23:07:52.749268 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.749323 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:52.749492 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:52.751634 (+ 2142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:52.751636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:52.751655 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:52.751772 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.751778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.753157 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.753160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.754645 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.754658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.757636 (+ 2978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.757647 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.757749 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.757771 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:07:52.757858 (+ 87us) client_negotiation.cc:240] Negotiation successful
0805 23:07:52.757934 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":32,"thread_start_us":81,"threads_started":1}
I20260805 23:07:52.758162 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.750011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.3.12.67:43739)
0805 23:07:52.751207 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.751223 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.751243 (+ 20us) server_negotiation.cc:407] Connection header received
0805 23:07:52.751319 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.751324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.751510 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.751666 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:52.751889 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.753019 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.756783 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.757502 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.757634 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.757772 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.757919 (+ 147us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.758013 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":156,"threads_started":1}
I20260805 23:07:52.759274 23373 tablet_service.cc:1467] Processing CreateTablet for tablet 4a919e83de7141eca428ccfbd05477cf (DEFAULT_TABLE table=test-table [id=b590f259e2634e9898aa500e7c8f192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:52.760224 23373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a919e83de7141eca428ccfbd05477cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:52.760546 23769 tablet_service.cc:1467] Processing CreateTablet for tablet 4a919e83de7141eca428ccfbd05477cf (DEFAULT_TABLE table=test-table [id=b590f259e2634e9898aa500e7c8f192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:52.760546 23577 tablet_service.cc:1467] Processing CreateTablet for tablet 4a919e83de7141eca428ccfbd05477cf (DEFAULT_TABLE table=test-table [id=b590f259e2634e9898aa500e7c8f192a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:52.761559 23577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a919e83de7141eca428ccfbd05477cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:52.761569 23769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a919e83de7141eca428ccfbd05477cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:52.766592 23898 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Bootstrap starting.
I20260805 23:07:52.768715 23899 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: Bootstrap starting.
I20260805 23:07:52.769029 23898 tablet_bootstrap.cc:654] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:52.769972 23898 log.cc:824] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:52.770751 23900 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: Bootstrap starting.
I20260805 23:07:52.771441 23899 tablet_bootstrap.cc:654] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:52.772198 23898 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: No bootstrap required, opened a new log
I20260805 23:07:52.772347 23899 log.cc:824] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:52.772459 23898 ts_tablet_manager.cc:1397] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:07:52.772966 23900 tablet_bootstrap.cc:654] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:52.774267 23900 log.cc:824] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:52.774552 23899 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: No bootstrap required, opened a new log
I20260805 23:07:52.774762 23899 ts_tablet_manager.cc:1397] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:07:52.776327 23900 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: No bootstrap required, opened a new log
I20260805 23:07:52.776518 23900 ts_tablet_manager.cc:1397] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:07:52.776376 23898 raft_consensus.cc:348] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.776683 23898 raft_consensus.cc:374] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:52.776738 23898 raft_consensus.cc:729] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764ee773f90e4b7a95d3b79d5bda74fb, State: Initialized, Role: FOLLOWER
I20260805 23:07:52.777202 23898 consensus_queue.cc:260] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.778235 23898 ts_tablet_manager.cc:1428] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:07:52.778594 23474 heartbeater.cc:503] Master 127.3.12.126:37039 was elected leader, sending a full tablet report...
I20260805 23:07:52.778640 23899 raft_consensus.cc:348] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.778908 23899 raft_consensus.cc:374] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:52.778975 23899 raft_consensus.cc:729] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b955aa5503b947af9c2d77c1ade0c5a0, State: Initialized, Role: FOLLOWER
I20260805 23:07:52.779450 23899 consensus_queue.cc:260] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.780150 23671 heartbeater.cc:503] Master 127.3.12.126:37039 was elected leader, sending a full tablet report...
I20260805 23:07:52.780470 23899 ts_tablet_manager.cc:1428] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 23:07:52.780370 23900 raft_consensus.cc:348] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.780628 23900 raft_consensus.cc:374] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:52.780675 23900 raft_consensus.cc:729] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7a779e84604edb9972cdbc2a5aa932, State: Initialized, Role: FOLLOWER
I20260805 23:07:52.781072 23900 consensus_queue.cc:260] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.782068 23900 ts_tablet_manager.cc:1428] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:07:52.782168 23877 heartbeater.cc:503] Master 127.3.12.126:37039 was elected leader, sending a full tablet report...
I20260805 23:07:52.820696 23906 raft_consensus.cc:482] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:07:52.820914 23906 raft_consensus.cc:504] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.822077 23906 leader_election.cc:288] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
I20260805 23:07:52.826781 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.822552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:42401 (local address 127.3.12.66:45055)
0805 23:07:52.822687 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.822693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.822904 (+ 211us) server_negotiation.cc:407] Connection header received
0805 23:07:52.823061 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.823064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.823131 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.823215 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:52.823536 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.824105 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.825712 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.826100 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.826236 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.826298 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.826540 (+ 242us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.826634 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:07:52.826785 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.822396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:43881 (local address 127.3.12.67:43739)
0805 23:07:52.822553 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.822558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.822849 (+ 291us) server_negotiation.cc:407] Connection header received
0805 23:07:52.822997 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.823003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.823071 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.823131 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:52.823522 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.824148 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.825659 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.826095 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.826226 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.826319 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.826534 (+ 215us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.826638 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 23:07:52.826869 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.822331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:45055 (local address 127.3.12.65:42401)
0805 23:07:52.822771 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.822783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:52.822939 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:52.823244 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:52.823247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:52.823289 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:52.823398 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.823403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.824217 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.824222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.825520 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.825530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.826236 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.826240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.826380 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.826420 (+ 40us) client_negotiation.cc:769] Sending connection context
0805 23:07:52.826516 (+ 96us) client_negotiation.cc:240] Negotiation successful
0805 23:07:52.826696 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":69,"spinlock_wait_cycles":768,"thread_start_us":156,"threads_started":1}
I20260805 23:07:52.826871 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.822324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43739 (local address 127.3.12.65:43881)
0805 23:07:52.822738 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.822751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:52.822852 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:52.823192 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:52.823198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:52.823218 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:52.823385 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.823391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.824313 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.824317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.825527 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:52.825537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.826234 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.826238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:52.826343 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.826416 (+ 73us) client_negotiation.cc:769] Sending connection context
0805 23:07:52.826515 (+ 99us) client_negotiation.cc:240] Negotiation successful
0805 23:07:52.826701 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":128,"spinlock_wait_cycles":22144,"thread_start_us":164,"threads_started":1}
I20260805 23:07:52.827577 23607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a919e83de7141eca428ccfbd05477cf" candidate_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" is_pre_election: true
I20260805 23:07:52.827586 23800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a919e83de7141eca428ccfbd05477cf" candidate_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7a779e84604edb9972cdbc2a5aa932" is_pre_election: true
I20260805 23:07:52.827916 23607 raft_consensus.cc:2454] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 764ee773f90e4b7a95d3b79d5bda74fb in term 0.
I20260805 23:07:52.827916 23800 raft_consensus.cc:2454] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 764ee773f90e4b7a95d3b79d5bda74fb in term 0.
I20260805 23:07:52.828408 23305 leader_election.cc:302] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb, b955aa5503b947af9c2d77c1ade0c5a0; no voters:
I20260805 23:07:52.828716 23906 raft_consensus.cc:2781] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:07:52.828809 23906 raft_consensus.cc:482] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:07:52.828855 23906 raft_consensus.cc:3037] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:52.829905 23906 raft_consensus.cc:504] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.830250 23906 leader_election.cc:288] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 1 election: Requested vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
I20260805 23:07:52.830591 23607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a919e83de7141eca428ccfbd05477cf" candidate_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b955aa5503b947af9c2d77c1ade0c5a0"
I20260805 23:07:52.830603 23800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a919e83de7141eca428ccfbd05477cf" candidate_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7a779e84604edb9972cdbc2a5aa932"
I20260805 23:07:52.830724 23607 raft_consensus.cc:3037] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:52.830734 23800 raft_consensus.cc:3037] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:52.831786 23800 raft_consensus.cc:2454] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 764ee773f90e4b7a95d3b79d5bda74fb in term 1.
I20260805 23:07:52.831792 23607 raft_consensus.cc:2454] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 764ee773f90e4b7a95d3b79d5bda74fb in term 1.
I20260805 23:07:52.832100 23310 leader_election.cc:302] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb, fe7a779e84604edb9972cdbc2a5aa932; no voters:
I20260805 23:07:52.832360 23906 raft_consensus.cc:2781] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:07:52.832600 23906 raft_consensus.cc:686] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 LEADER]: Becoming Leader. State: Replica: 764ee773f90e4b7a95d3b79d5bda74fb, State: Running, Role: LEADER
I20260805 23:07:52.832973 23906 consensus_queue.cc:237] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:52.836228 23191 catalog_manager.cc:5495] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 764ee773f90e4b7a95d3b79d5bda74fb (127.3.12.65). New cstate: current_term: 1 leader_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } health_report { overall_health: HEALTHY } } }
I20260805 23:07:52.852806 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.849044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.3.12.65:35869)
0805 23:07:52.849194 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:52.849198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:52.849212 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:07:52.849374 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:52.849377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:52.849443 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:52.849554 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:07:52.849827 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.850367 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.851240 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:52.851476 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:52.851586 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:52.851927 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:07:52.852422 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:07:52.852552 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:52.852612 (+ 60us) server_negotiation.cc:299] Negotiation successful
0805 23:07:52.852669 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:07:52.858631 23607 raft_consensus.cc:1261] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 1 FOLLOWER]: Refusing update from remote peer 764ee773f90e4b7a95d3b79d5bda74fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:07:52.858690 23800 raft_consensus.cc:1261] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 1 FOLLOWER]: Refusing update from remote peer 764ee773f90e4b7a95d3b79d5bda74fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:07:52.859380 23912 consensus_queue.cc:1035] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:07:52.859577 23906 consensus_queue.cc:1035] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:07:52.868122 23920 mvcc.cc:205] Tried to move back new op lower bound from 7315338333621587968 to 7315338333526659072. Current Snapshot: MvccSnapshot[applied={T|T < 7315338333621587968}]
I20260805 23:07:52.876361 23191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:07:52.876560 23191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:07:52.879087 23191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Sending DeleteTablet for 3 replicas of tablet 4a919e83de7141eca428ccfbd05477cf
I20260805 23:07:52.879938 23577 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a919e83de7141eca428ccfbd05477cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:52 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:44882
I20260805 23:07:52.879937 23373 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a919e83de7141eca428ccfbd05477cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:52 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:49298
I20260805 23:07:52.880024 23769 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a919e83de7141eca428ccfbd05477cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:52 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:34264
I20260805 23:07:52.880467 23934 tablet_replica.cc:330] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: stopping tablet replica
I20260805 23:07:52.880688 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 23281
I20260805 23:07:52.880693 23933 tablet_replica.cc:330] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: stopping tablet replica
I20260805 23:07:52.887686 23937 tablet_replica.cc:330] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: stopping tablet replica
I20260805 23:07:52.888111 23937 raft_consensus.cc:2229] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:07:52.880827 23934 raft_consensus.cc:2229] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:07:52.888532 23937 raft_consensus.cc:2258] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:07:52.888532 23934 raft_consensus.cc:2258] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:07:52.890202 23934 ts_tablet_manager.cc:1905] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:07:52.892637 23934 ts_tablet_manager.cc:1918] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:07:52.892724 23934 log.cc:1196] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/4a919e83de7141eca428ccfbd05477cf
I20260805 23:07:52.893157 23934 ts_tablet_manager.cc:1939] T 4a919e83de7141eca428ccfbd05477cf P b955aa5503b947af9c2d77c1ade0c5a0: Deleting consensus metadata
W20260805 23:07:52.894181 23178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:35869 (error 108)
I20260805 23:07:52.894367 23937 ts_tablet_manager.cc:1905] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:07:52.894470 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 23484
I20260805 23:07:52.894563 23174 catalog_manager.cc:4841] TS b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055): tablet 4a919e83de7141eca428ccfbd05477cf (table test-table [id=b590f259e2634e9898aa500e7c8f192a]) successfully deleted
I20260805 23:07:52.896780 23937 ts_tablet_manager.cc:1918] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:07:52.896885 23937 log.cc:1196] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/4a919e83de7141eca428ccfbd05477cf
I20260805 23:07:52.897261 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:52.896866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:35869 (local address 127.0.0.1:49316)
0805 23:07:52.897073 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:52.897152 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:35869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260805 23:07:52.897270 23937 ts_tablet_manager.cc:1939] T 4a919e83de7141eca428ccfbd05477cf P fe7a779e84604edb9972cdbc2a5aa932: Deleting consensus metadata
W20260805 23:07:52.897568 23178 catalog_manager.cc:4584] TS 764ee773f90e4b7a95d3b79d5bda74fb (127.3.12.65:35869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a919e83de7141eca428ccfbd05477cf: Network error: Client connection negotiation failed: client connection to 127.3.12.65:35869: connect: Connection refused (error 111)
I20260805 23:07:52.898550 23178 catalog_manager.cc:4841] TS fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739): tablet 4a919e83de7141eca428ccfbd05477cf (table test-table [id=b590f259e2634e9898aa500e7c8f192a]) successfully deleted
I20260805 23:07:52.905900 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 23679
I20260805 23:07:52.917709 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 23152
I20260805 23:07:52.929229 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37039
--webserver_interface=127.3.12.126
--webserver_port=41609
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:37039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:53.043401 23941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:53.043674 23941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:53.043716 23941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:53.047818 23941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:53.047878 23941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:53.047899 23941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:53.047914 23941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:53.047930 23941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:53.053330 23941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:37039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=41609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:53.054426 23941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:53.055518 23941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:53.069721 23951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:53.069716 23949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:53.069844 23941 server_base.cc:1034] running on GCE node
W20260805 23:07:53.069720 23953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:53.070664 23941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:53.071880 23941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:53.073086 23941 hybrid_clock.cc:648] HybridClock initialized: now 1785971273073039 us; error 58 us; skew 500 ppm
Aug 05 23:07:53 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:53.075881 23941 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:07:53.076894 23941 webserver.cc:460] Webserver started at http://127.3.12.126:41609/ using document root <none> and password file <none>
I20260805 23:07:53.077508 23941 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:53.077626 23941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:53.081606 23941 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:07:53.084005 23963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:53.085520 23941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:07:53.085668 23941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "628c3b4d034f45829c11dc189afb0163"
format_stamp: "Formatted at 2026-08-05 23:07:51 on dist-test-slave-2d19"
server_key: "e6e4d0ad104ff34ae4167fe41b914f81"
server_key_iv: "cec4e2cb02c22c4eb9de3407f402d6a5"
server_key_version: "encryptionkey@0"
I20260805 23:07:53.086308 23941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:53.110270 23941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:53.113703 23941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:53.113946 23941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:53.121421 23941 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:37039
I20260805 23:07:53.121805 24048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:37039 every 8 connection(s)
I20260805 23:07:53.122582 23941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:07:53.126614 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 23941
I20260805 23:07:53.127147 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:35869
--local_ip_for_outbound_sockets=127.3.12.65
--tserver_master_addrs=127.3.12.126:37039
--webserver_port=46419
--webserver_interface=127.3.12.65
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:07:53.127616 24050 sys_catalog.cc:263] Verifying existing consensus state
I20260805 23:07:53.129995 24050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Bootstrap starting.
I20260805 23:07:53.147552 24050 log.cc:824] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:53.156953 24050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:07:53.157356 24050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Bootstrap complete.
I20260805 23:07:53.161516 24050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:53.161895 24050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628c3b4d034f45829c11dc189afb0163, State: Initialized, Role: FOLLOWER
I20260805 23:07:53.162429 24050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:53.162549 24050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:07:53.162637 24050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:07:53.162752 24050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 1 FOLLOWER]: Advancing to term 2
I20260805 23:07:53.164707 24050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:53.165046 24050 leader_election.cc:302] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 628c3b4d034f45829c11dc189afb0163; no voters:
I20260805 23:07:53.165382 24050 leader_election.cc:288] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260805 23:07:53.165472 24056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 23:07:53.165673 24056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 2 LEADER]: Becoming Leader. State: Replica: 628c3b4d034f45829c11dc189afb0163, State: Running, Role: LEADER
I20260805 23:07:53.165953 24056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:53.166615 24050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:07:53.166914 24056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 628c3b4d034f45829c11dc189afb0163. Latest consensus state: current_term: 2 leader_uuid: "628c3b4d034f45829c11dc189afb0163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } } }
I20260805 23:07:53.167039 24056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:53.167403 24061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:07:53.167459 24057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "628c3b4d034f45829c11dc189afb0163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } } }
I20260805 23:07:53.167716 24057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:53.172210 24061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:07:53.173096 24061 catalog_manager.cc:1259] Loaded cluster ID: 4a73ef5de62f4fc4b31cddca8ca60ec4
I20260805 23:07:53.173162 24061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:07:53.174606 24061 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260805 23:07:53.179502 24061 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260805 23:07:53.185144 24061 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:07:53.186103 24061 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Loaded TSK: 0
I20260805 23:07:53.187290 24061 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 23:07:53.249519 24054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:53.249804 24054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:53.249893 24054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:53.254063 24054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:53.254148 24054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:53.254240 24054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:07:53.259506 24054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:35869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=46419
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:53.260566 24054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:53.261602 24054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:53.272418 24093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:53.272373 24089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:53.272373 24091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:53.272686 24054 server_base.cc:1034] running on GCE node
I20260805 23:07:53.273242 24054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:53.274000 24054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:53.275206 24054 hybrid_clock.cc:648] HybridClock initialized: now 1785971273275181 us; error 32 us; skew 500 ppm
Aug 05 23:07:53 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:53.278688 24054 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:07:53.279752 24054 webserver.cc:460] Webserver started at http://127.3.12.65:46419/ using document root <none> and password file <none>
I20260805 23:07:53.280469 24054 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:53.280588 24054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:53.285756 24054 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260805 23:07:53.288542 24102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:53.290270 24054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260805 23:07:53.290464 24054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "764ee773f90e4b7a95d3b79d5bda74fb"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2fc060d10a33dde404787ac275f828f4"
server_key_iv: "5cdbb8dcacc37a76ca75734edf758567"
server_key_version: "encryptionkey@0"
I20260805 23:07:53.291164 24054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:53.348424 24054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:53.351783 24054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:53.352063 24054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:53.352766 24054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:53.354014 24111 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260805 23:07:53.358317 24054 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260805 23:07:53.358423 24054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260805 23:07:53.358518 24054 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260805 23:07:53.360288 24054 ts_tablet_manager.cc:610] Registered 1 tablets
I20260805 23:07:53.360391 24054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260805 23:07:53.360692 24111 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Bootstrap starting.
I20260805 23:07:53.370914 24054 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:35869
I20260805 23:07:53.370956 24277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:35869 every 8 connection(s)
I20260805 23:07:53.371858 24054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:07:53.375979 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 24054
I20260805 23:07:53.376540 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:45055
--local_ip_for_outbound_sockets=127.3.12.66
--tserver_master_addrs=127.3.12.126:37039
--webserver_port=37899
--webserver_interface=127.3.12.66
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:07:53.382905 24111 log.cc:824] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:07:53 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:53.392005 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.374216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:50095 (local address 127.3.12.126:37039)
0805 23:07:53.374695 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.374723 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.374971 (+ 248us) server_negotiation.cc:407] Connection header received
0805 23:07:53.376142 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.376157 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.376535 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.376907 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:53.378140 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.379649 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.381298 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.381782 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.381972 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.385322 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:53.385329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:53.385346 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:53.385382 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:53.387687 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.388448 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.388460 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.388466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.388625 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.389206 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.389210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.389212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.389780 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:53.389943 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.390752 (+ 809us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.391082 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":138,"threads_started":1}
I20260805 23:07:53.392056 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.374153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:50095)
0805 23:07:53.374782 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:53.374836 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:53.375672 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:53.377053 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:53.377062 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:53.377469 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:53.377948 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.377970 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.379859 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.379863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.381157 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.381170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.381950 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.381963 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.382173 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.382900 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:53.382918 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:53.385073 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:53.387837 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:53.387845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:53.387863 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:53.388312 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:53.388751 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:53.388757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:53.388760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:53.388974 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:53.389931 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:53.389944 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:53.390318 (+ 374us) client_negotiation.cc:769] Sending connection context
0805 23:07:53.390582 (+ 264us) client_negotiation.cc:240] Negotiation successful
0805 23:07:53.391115 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":158,"threads_started":1}
I20260805 23:07:53.392691 24111 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:07:53.393133 24111 tablet_bootstrap.cc:492] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Bootstrap complete.
I20260805 23:07:53.394598 24111 ts_tablet_manager.cc:1397] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.013s
I20260805 23:07:53.395308 24278 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37039
I20260805 23:07:53.395543 24278 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:53.396119 24278 heartbeater.cc:511] Master 127.3.12.126:37039 requested a full tablet report, sending...
I20260805 23:07:53.397522 24111 raft_consensus.cc:348] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.397957 24111 raft_consensus.cc:729] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764ee773f90e4b7a95d3b79d5bda74fb, State: Initialized, Role: FOLLOWER
I20260805 23:07:53.398459 24111 consensus_queue.cc:260] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.398991 23988 ts_manager.cc:194] Registered new tserver with Master: 764ee773f90e4b7a95d3b79d5bda74fb (127.3.12.65:35869)
I20260805 23:07:53.399535 24111 ts_tablet_manager.cc:1428] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260805 23:07:53.402689 23988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:50095
I20260805 23:07:53.405391 24278 heartbeater.cc:503] Master 127.3.12.126:37039 was elected leader, sending a full tablet report...
Aug 05 23:07:53 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:07:53.414296 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.402028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:35869 (local address 127.0.0.1:49320)
0805 23:07:53.402856 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:53.402900 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:53.403038 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:53.403624 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:53.403631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:53.403998 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:53.404111 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.404117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.405155 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.405162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.405891 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.405904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.406657 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.406661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.406775 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.407495 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:53.407511 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:53.409108 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:53.412010 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:53.412021 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:53.412027 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:53.412345 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:53.412670 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:53.412674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:53.412676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:53.412803 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:53.413532 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:53.413538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:53.413702 (+ 164us) client_negotiation.cc:769] Sending connection context
0805 23:07:53.413880 (+ 178us) client_negotiation.cc:240] Negotiation successful
0805 23:07:53.414105 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":128,"threads_started":1}
I20260805 23:07:53.414315 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.402136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.3.12.65:35869)
0805 23:07:53.402534 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.402546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.402986 (+ 440us) server_negotiation.cc:407] Connection header received
0805 23:07:53.403207 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.403213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.403409 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.403544 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:53.404238 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.405028 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.406030 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.406517 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.406661 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.409251 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:53.409267 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:53.409278 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:53.409300 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:53.411872 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.412462 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.412468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.412470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.412561 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.412933 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.412938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.412941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.413307 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:53.413471 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.413996 (+ 525us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.414166 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":302,"thread_start_us":119,"threads_started":1}
I20260805 23:07:53.415360 24182 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a919e83de7141eca428ccfbd05477cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:52 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:49320
I20260805 23:07:53.419497 24290 tablet_replica.cc:330] stopping tablet replica
I20260805 23:07:53.419839 24290 raft_consensus.cc:2229] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:07:53.420156 24290 raft_consensus.cc:2258] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:07:53.421478 24290 ts_tablet_manager.cc:1905] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:07:53.427421 24290 ts_tablet_manager.cc:1918] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:07:53.427559 24290 log.cc:1196] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/4a919e83de7141eca428ccfbd05477cf
I20260805 23:07:53.427932 24290 ts_tablet_manager.cc:1939] T 4a919e83de7141eca428ccfbd05477cf P 764ee773f90e4b7a95d3b79d5bda74fb: Deleting consensus metadata
I20260805 23:07:53.430295 23969 catalog_manager.cc:4841] TS 764ee773f90e4b7a95d3b79d5bda74fb (127.3.12.65:35869): tablet 4a919e83de7141eca428ccfbd05477cf (table test-table [id=b590f259e2634e9898aa500e7c8f192a]) successfully deleted
W20260805 23:07:53.513474 24285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:53.513769 24285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:53.513815 24285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:53.518059 24285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:53.518126 24285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:53.518219 24285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:07:53.523855 24285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:45055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=37899
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:53.525156 24285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:53.526242 24285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:53.536387 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:53.536693 24285 server_base.cc:1034] running on GCE node
W20260805 23:07:53.536387 24299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:53.536399 24303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:53.537204 24285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:53.537930 24285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:53.539129 24285 hybrid_clock.cc:648] HybridClock initialized: now 1785971273539097 us; error 44 us; skew 500 ppm
Aug 05 23:07:53 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:53.542363 24285 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:07:53.543644 24285 webserver.cc:460] Webserver started at http://127.3.12.66:37899/ using document root <none> and password file <none>
I20260805 23:07:53.544471 24285 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:53.544611 24285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:53.550436 24285 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260805 23:07:53.553385 24313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:53.555261 24285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:07:53.555419 24285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "b955aa5503b947af9c2d77c1ade0c5a0"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "9e75a607616bd5a536ca0c2276742b28"
server_key_iv: "1e3ce28fc91bc2867adc16c59756a05a"
server_key_version: "encryptionkey@0"
I20260805 23:07:53.556326 24285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:53.576767 24285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:53.581099 24285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:53.581341 24285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:53.582237 24285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:53.583557 24285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:53.583634 24285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:53.583702 24285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:53.583741 24285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:53.597657 24285 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:45055
I20260805 23:07:53.597707 24481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:45055 every 8 connection(s)
I20260805 23:07:53.599198 24285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:07:53.606693 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 24285
I20260805 23:07:53.607156 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:43739
--local_ip_for_outbound_sockets=127.3.12.67
--tserver_master_addrs=127.3.12.126:37039
--webserver_port=35533
--webserver_interface=127.3.12.67
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 23:07:53 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:53.617976 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.601530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:57405 (local address 127.3.12.126:37039)
0805 23:07:53.601685 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.601689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.602894 (+ 1205us) server_negotiation.cc:407] Connection header received
0805 23:07:53.604284 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.604288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.604358 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.604436 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:53.605849 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.606451 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.608391 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.608810 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.608936 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.612636 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:53.612643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:53.612647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:53.612688 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:53.614770 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.615562 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.615568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.615571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.615675 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.616359 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.616367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.616370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.616660 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:53.616786 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.617534 (+ 748us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.617709 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260805 23:07:53.619350 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.602153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:57405)
0805 23:07:53.602719 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:53.602764 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:53.604029 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:53.604772 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:53.604779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:53.605144 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:53.605695 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.605710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.606626 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.606632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.608230 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.608245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.609007 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.609013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.609268 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.610073 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:53.610102 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:53.612371 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:53.614924 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:53.614934 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:53.614954 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:53.615418 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:53.615795 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:53.615801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:53.615805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:53.616018 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:53.616806 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:53.616815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:53.617201 (+ 386us) client_negotiation.cc:769] Sending connection context
0805 23:07:53.617461 (+ 260us) client_negotiation.cc:240] Negotiation successful
0805 23:07:53.617910 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":167,"threads_started":1}
I20260805 23:07:53.620759 24482 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37039
I20260805 23:07:53.621037 24482 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:53.621659 24482 heartbeater.cc:511] Master 127.3.12.126:37039 requested a full tablet report, sending...
I20260805 23:07:53.622887 23988 ts_manager.cc:194] Registered new tserver with Master: b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
I20260805 23:07:53.623540 23988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:57405
W20260805 23:07:53.727947 24485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:53.728190 24485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:53.728232 24485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:53.732165 24485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:53.732281 24485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:53.732372 24485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:07:53.737593 24485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:43739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=35533
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:53.738629 24485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:53.739661 24485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:53.749483 24493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:53.749511 24490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:53.749511 24491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:53.749569 24485 server_base.cc:1034] running on GCE node
I20260805 23:07:53.750197 24485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:53.750828 24485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:53.752152 24485 hybrid_clock.cc:648] HybridClock initialized: now 1785971273752133 us; error 27 us; skew 500 ppm
Aug 05 23:07:53 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:53.755391 24485 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:07:53.756340 24485 webserver.cc:460] Webserver started at http://127.3.12.67:35533/ using document root <none> and password file <none>
I20260805 23:07:53.756875 24485 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:53.756945 24485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:53.760748 24485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:07:53.762874 24506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:53.764236 24485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:07:53.764393 24485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "fe7a779e84604edb9972cdbc2a5aa932"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2d9bbd0f9a6b5dfa446c82cba1fd7495"
server_key_iv: "5f6c198befceee7276e95eb36c4febe6"
server_key_version: "encryptionkey@0"
I20260805 23:07:53.764897 24485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:53.787551 24485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:53.790906 24485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:53.791124 24485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:53.791764 24485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:53.792703 24485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:53.792760 24485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:53.792837 24485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:53.792874 24485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:53.802066 24485 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:43739
I20260805 23:07:53.802151 24678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:43739 every 8 connection(s)
I20260805 23:07:53.803143 24485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:07:53.805904 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 24485
Aug 05 23:07:53 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971273, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:07:53.815438 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.806623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.3.12.126:37039)
0805 23:07:53.806940 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.806944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.807066 (+ 122us) server_negotiation.cc:407] Connection header received
0805 23:07:53.807212 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.807214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.807260 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.807332 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:53.808334 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.808862 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.809859 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.810158 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.810233 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.811841 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:53.811845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:53.811847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:53.811875 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:53.813491 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.813932 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.813939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.813942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.814035 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.814335 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.814340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.814342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.814592 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:53.814799 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.815043 (+ 244us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.815205 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
I20260805 23:07:53.818236 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.805070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:52577 (local address 127.3.12.126:37039)
0805 23:07:53.805218 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.805223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.806134 (+ 911us) server_negotiation.cc:407] Connection header received
0805 23:07:53.807901 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.807903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.807935 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.807995 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:53.809374 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.809888 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.811528 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.811735 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.811869 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.814449 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:53.814460 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:53.814463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:53.814484 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:53.816075 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.816740 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.816743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.816745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.816834 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:53.817174 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:53.817177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:53.817179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:53.817386 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:53.817506 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.818038 (+ 532us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.818131 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260805 23:07:53.819087 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.805417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:52577)
0805 23:07:53.805952 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:53.806022 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:53.807670 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:53.808203 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:53.808216 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:53.808626 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:53.809222 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.809243 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.810014 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.810016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.811378 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.811394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.811934 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.811937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.812081 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.812717 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:53.812755 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:53.814299 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:53.816185 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:53.816191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:53.816207 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:53.816606 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:53.816925 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:53.816928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:53.816929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:53.817064 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:53.817504 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:53.817510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:53.817806 (+ 296us) client_negotiation.cc:769] Sending connection context
0805 23:07:53.817958 (+ 152us) client_negotiation.cc:240] Negotiation successful
0805 23:07:53.818220 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":137,"threads_started":1}
I20260805 23:07:53.820130 24679 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37039
I20260805 23:07:53.820411 24679 heartbeater.cc:464] Registering TS with master...
I20260805 23:07:53.820904 24679 heartbeater.cc:511] Master 127.3.12.126:37039 requested a full tablet report, sending...
I20260805 23:07:53.821908 23988 ts_manager.cc:194] Registered new tserver with Master: fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739)
I20260805 23:07:53.822482 23988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:52577
I20260805 23:07:53.829432 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:07:53.833767 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.830068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.3.12.126:37039)
0805 23:07:53.830221 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.830226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.830415 (+ 189us) server_negotiation.cc:407] Connection header received
0805 23:07:53.830555 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.830557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.830652 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.830740 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:07:53.830975 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.831491 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.832406 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.832634 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.832722 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.833014 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:07:53.833391 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:07:53.833487 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.833545 (+ 58us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.833595 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 23:07:53.834396 23988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:07:53.837735 23988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:07:53.845746 24182 tablet_service.cc:1467] Processing CreateTablet for tablet 5f11ab6f09394fe3813ad3aadd999292 (DEFAULT_TABLE table=test-table [id=4c29a54b9bf047a1b4b94af243ee6ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:53.846421 24182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f11ab6f09394fe3813ad3aadd999292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:53.849442 24111 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Bootstrap starting.
I20260805 23:07:53.850385 24111 tablet_bootstrap.cc:654] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:53.850553 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.844592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43739 (local address 127.0.0.1:34278)
0805 23:07:53.844756 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:53.844771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:53.844854 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:53.845939 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:53.845945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:53.845967 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:53.846161 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.846172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.847441 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.847446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.848679 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.848691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.850078 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.850082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.850186 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.850230 (+ 44us) client_negotiation.cc:769] Sending connection context
0805 23:07:53.850315 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 23:07:53.850401 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":50,"spinlock_wait_cycles":2304}
I20260805 23:07:53.850567 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.844763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.3.12.67:43739)
0805 23:07:53.845552 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.845561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.845579 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:07:53.845639 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.845645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.845790 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.846007 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:53.846310 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.847300 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.848829 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.849940 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.850081 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.850240 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.850347 (+ 107us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.850437 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":224,"threads_started":1}
I20260805 23:07:53.851831 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.844826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:45055 (local address 127.0.0.1:44886)
0805 23:07:53.845203 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:53.845219 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:53.845309 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:53.845969 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:53.845972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:53.846002 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:53.846214 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.846225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.847754 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.847758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.848894 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.848904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.851402 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.851405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.851495 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.851517 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:07:53.851600 (+ 83us) client_negotiation.cc:240] Negotiation successful
0805 23:07:53.851694 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":57,"spinlock_wait_cycles":768,"thread_start_us":140,"threads_started":1}
I20260805 23:07:53.851876 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.845107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.3.12.66:45055)
0805 23:07:53.845441 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.845453 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.845480 (+ 27us) server_negotiation.cc:407] Connection header received
0805 23:07:53.845561 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.845566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.845784 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.846313 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:53.846476 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.847606 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.849030 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.851240 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.851396 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.851541 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.851645 (+ 104us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.851738 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20260805 23:07:53.852751 24111 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: No bootstrap required, opened a new log
I20260805 23:07:53.852852 24111 ts_tablet_manager.cc:1397] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:07:53.853372 24111 raft_consensus.cc:348] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.853510 24111 raft_consensus.cc:374] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:53.853557 24111 raft_consensus.cc:729] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764ee773f90e4b7a95d3b79d5bda74fb, State: Initialized, Role: FOLLOWER
I20260805 23:07:53.853751 24111 consensus_queue.cc:260] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.854148 24111 ts_tablet_manager.cc:1428] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:07:53.853760 24582 tablet_service.cc:1467] Processing CreateTablet for tablet 5f11ab6f09394fe3813ad3aadd999292 (DEFAULT_TABLE table=test-table [id=4c29a54b9bf047a1b4b94af243ee6ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:53.854926 24582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f11ab6f09394fe3813ad3aadd999292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:53.854784 24377 tablet_service.cc:1467] Processing CreateTablet for tablet 5f11ab6f09394fe3813ad3aadd999292 (DEFAULT_TABLE table=test-table [id=4c29a54b9bf047a1b4b94af243ee6ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:07:53.855832 24377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f11ab6f09394fe3813ad3aadd999292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:07:53.863384 24697 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: Bootstrap starting.
I20260805 23:07:53.863456 24698 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: Bootstrap starting.
I20260805 23:07:53.865820 24697 tablet_bootstrap.cc:654] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:53.866602 24698 tablet_bootstrap.cc:654] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: Neither blocks nor log segments found. Creating new log.
I20260805 23:07:53.867256 24697 log.cc:824] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:53.867408 24698 log.cc:824] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:53.869560 24697 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: No bootstrap required, opened a new log
I20260805 23:07:53.869776 24697 ts_tablet_manager.cc:1397] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260805 23:07:53.870033 24698 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: No bootstrap required, opened a new log
I20260805 23:07:53.870235 24698 ts_tablet_manager.cc:1397] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260805 23:07:53.872798 24697 raft_consensus.cc:348] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.873051 24697 raft_consensus.cc:374] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:53.873111 24697 raft_consensus.cc:729] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7a779e84604edb9972cdbc2a5aa932, State: Initialized, Role: FOLLOWER
W20260805 23:07:53.873091 24280 tablet.cc:2369] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:07:53.873131 24698 raft_consensus.cc:348] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.873389 24698 raft_consensus.cc:374] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:07:53.873473 24698 raft_consensus.cc:729] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b955aa5503b947af9c2d77c1ade0c5a0, State: Initialized, Role: FOLLOWER
I20260805 23:07:53.873636 24697 consensus_queue.cc:260] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.873929 24698 consensus_queue.cc:260] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.874455 24679 heartbeater.cc:503] Master 127.3.12.126:37039 was elected leader, sending a full tablet report...
I20260805 23:07:53.874689 24697 ts_tablet_manager.cc:1428] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260805 23:07:53.874917 24698 ts_tablet_manager.cc:1428] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:07:53.875279 24482 heartbeater.cc:503] Master 127.3.12.126:37039 was elected leader, sending a full tablet report...
I20260805 23:07:53.960448 24287 raft_consensus.cc:482] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:07:53.960717 24287 raft_consensus.cc:504] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.962088 24287 leader_election.cc:288] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
I20260805 23:07:53.966528 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.962451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:43223 (local address 127.3.12.67:43739)
0805 23:07:53.962604 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.962609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.962801 (+ 192us) server_negotiation.cc:407] Connection header received
0805 23:07:53.963071 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.963074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.963145 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.963237 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:53.963553 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.964060 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.965244 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.965887 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.966020 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.966085 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.966325 (+ 240us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.966397 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:07:53.966563 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.962295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43739 (local address 127.3.12.65:43223)
0805 23:07:53.962693 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:53.962705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:53.962858 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:53.963252 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:53.963254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:53.963299 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:53.963440 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.963446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.964182 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.964186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.965110 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.965121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.966015 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.966019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.966127 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.966196 (+ 69us) client_negotiation.cc:769] Sending connection context
0805 23:07:53.966304 (+ 108us) client_negotiation.cc:240] Negotiation successful
0805 23:07:53.966381 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":37,"spinlock_wait_cycles":5376,"thread_start_us":130,"threads_started":1}
I20260805 23:07:53.966603 24705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.962403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:45055 (local address 127.3.12.65:43983)
0805 23:07:53.962774 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:53.962786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:53.962914 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:53.963249 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:53.963252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:53.963297 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:07:53.963421 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.963426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.964165 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.964172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.965156 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:53.965169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.966148 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.966151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:53.966243 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.966269 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 23:07:53.966344 (+ 75us) client_negotiation.cc:240] Negotiation successful
0805 23:07:53.966427 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":52,"spinlock_wait_cycles":33536,"thread_start_us":69,"threads_started":1}
I20260805 23:07:53.966600 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:53.962501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:43983 (local address 127.3.12.66:45055)
0805 23:07:53.962690 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:53.962695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:53.962889 (+ 194us) server_negotiation.cc:407] Connection header received
0805 23:07:53.963074 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:53.963077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:53.963132 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:53.963237 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:07:53.963538 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.964045 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.965341 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:53.965940 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:53.966034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:53.966085 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:53.966381 (+ 296us) server_negotiation.cc:299] Negotiation successful
0805 23:07:53.966461 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260805 23:07:53.967278 24412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f11ab6f09394fe3813ad3aadd999292" candidate_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" is_pre_election: true
I20260805 23:07:53.967471 24610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f11ab6f09394fe3813ad3aadd999292" candidate_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7a779e84604edb9972cdbc2a5aa932" is_pre_election: true
I20260805 23:07:53.967641 24412 raft_consensus.cc:2454] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 764ee773f90e4b7a95d3b79d5bda74fb in term 0.
I20260805 23:07:53.967816 24610 raft_consensus.cc:2454] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 764ee773f90e4b7a95d3b79d5bda74fb in term 0.
I20260805 23:07:53.968261 24107 leader_election.cc:302] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb, b955aa5503b947af9c2d77c1ade0c5a0, fe7a779e84604edb9972cdbc2a5aa932; no voters:
I20260805 23:07:53.968568 24287 raft_consensus.cc:2781] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:07:53.968657 24287 raft_consensus.cc:482] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:07:53.968701 24287 raft_consensus.cc:3037] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:53.969842 24287 raft_consensus.cc:504] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.970211 24287 leader_election.cc:288] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 1 election: Requested vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
I20260805 23:07:53.970566 24610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f11ab6f09394fe3813ad3aadd999292" candidate_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7a779e84604edb9972cdbc2a5aa932"
I20260805 23:07:53.970647 24412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f11ab6f09394fe3813ad3aadd999292" candidate_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b955aa5503b947af9c2d77c1ade0c5a0"
I20260805 23:07:53.970698 24610 raft_consensus.cc:3037] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:53.970777 24412 raft_consensus.cc:3037] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:07:53.972047 24412 raft_consensus.cc:2454] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 764ee773f90e4b7a95d3b79d5bda74fb in term 1.
I20260805 23:07:53.972047 24610 raft_consensus.cc:2454] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 764ee773f90e4b7a95d3b79d5bda74fb in term 1.
I20260805 23:07:53.972374 24107 leader_election.cc:302] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb, fe7a779e84604edb9972cdbc2a5aa932; no voters:
I20260805 23:07:53.972621 24287 raft_consensus.cc:2781] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:07:53.972795 24287 raft_consensus.cc:686] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 LEADER]: Becoming Leader. State: Replica: 764ee773f90e4b7a95d3b79d5bda74fb, State: Running, Role: LEADER
I20260805 23:07:53.973057 24287 consensus_queue.cc:237] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:53.976150 23988 catalog_manager.cc:5495] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 764ee773f90e4b7a95d3b79d5bda74fb (127.3.12.65). New cstate: current_term: 1 leader_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } health_report { overall_health: HEALTHY } } }
I20260805 23:07:54.004415 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.000467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.3.12.65:35869)
0805 23:07:54.000913 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:54.000921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:54.000936 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:07:54.001022 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:54.001024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:54.001084 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:54.001209 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:07:54.001435 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:54.001981 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:54.002823 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:54.003080 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:54.003176 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:54.003465 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:07:54.003928 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:07:54.004024 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:54.004096 (+ 72us) server_negotiation.cc:299] Negotiation successful
0805 23:07:54.004147 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":185,"threads_started":1}
I20260805 23:07:54.008715 24610 raft_consensus.cc:1261] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 1 FOLLOWER]: Refusing update from remote peer 764ee773f90e4b7a95d3b79d5bda74fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:07:54.008715 24412 raft_consensus.cc:1261] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 1 FOLLOWER]: Refusing update from remote peer 764ee773f90e4b7a95d3b79d5bda74fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:07:54.009541 24712 consensus_queue.cc:1035] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:07:54.009748 24287 consensus_queue.cc:1035] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:07:54.017627 24715 mvcc.cc:205] Tried to move back new op lower bound from 7315338338332704768 to 7315338338196332544. Current Snapshot: MvccSnapshot[applied={T|T < 7315338338332704768}]
I20260805 23:07:54.019672 24717 mvcc.cc:205] Tried to move back new op lower bound from 7315338338332704768 to 7315338338196332544. Current Snapshot: MvccSnapshot[applied={T|T < 7315338338332704768}]
I20260805 23:07:54.027619 23987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:07:54.027813 23987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:07:54.030364 23987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Sending DeleteTablet for 3 replicas of tablet 5f11ab6f09394fe3813ad3aadd999292
I20260805 23:07:54.031136 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 24054
I20260805 23:07:54.031193 24377 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f11ab6f09394fe3813ad3aadd999292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:54 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:44886
I20260805 23:07:54.031338 24582 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f11ab6f09394fe3813ad3aadd999292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:07:54 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:34278
I20260805 23:07:54.036799 24726 tablet_replica.cc:330] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: stopping tablet replica
I20260805 23:07:54.037320 24726 raft_consensus.cc:2229] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:07:54.037806 24726 raft_consensus.cc:2258] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:07:54.040767 24726 ts_tablet_manager.cc:1905] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:07:54.043012 24727 tablet_replica.cc:330] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: stopping tablet replica
I20260805 23:07:54.043346 24727 raft_consensus.cc:2229] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:07:54.043689 24726 ts_tablet_manager.cc:1918] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:07:54.043692 24727 raft_consensus.cc:2258] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:07:54.043824 24726 log.cc:1196] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/5f11ab6f09394fe3813ad3aadd999292
I20260805 23:07:54.044191 24726 ts_tablet_manager.cc:1939] T 5f11ab6f09394fe3813ad3aadd999292 P b955aa5503b947af9c2d77c1ade0c5a0: Deleting consensus metadata
I20260805 23:07:54.045409 24727 ts_tablet_manager.cc:1905] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:07:54.045450 23964 catalog_manager.cc:4841] TS b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055): tablet 5f11ab6f09394fe3813ad3aadd999292 (table test-table [id=4c29a54b9bf047a1b4b94af243ee6ca6]) successfully deleted
I20260805 23:07:54.048086 24727 ts_tablet_manager.cc:1918] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:07:54.048202 24727 log.cc:1196] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/5f11ab6f09394fe3813ad3aadd999292
I20260805 23:07:54.048555 24727 ts_tablet_manager.cc:1939] T 5f11ab6f09394fe3813ad3aadd999292 P fe7a779e84604edb9972cdbc2a5aa932: Deleting consensus metadata
I20260805 23:07:54.049841 23969 catalog_manager.cc:4841] TS fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739): tablet 5f11ab6f09394fe3813ad3aadd999292 (table test-table [id=4c29a54b9bf047a1b4b94af243ee6ca6]) successfully deleted
W20260805 23:07:54.050232 23969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:35869 (error 108)
I20260805 23:07:54.050536 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 24285
I20260805 23:07:54.055781 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.055459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:35869 (local address 127.0.0.1:49344)
0805 23:07:54.055623 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.055697 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:35869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260805 23:07:54.056023 23969 catalog_manager.cc:4584] TS 764ee773f90e4b7a95d3b79d5bda74fb (127.3.12.65:35869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f11ab6f09394fe3813ad3aadd999292: Network error: Client connection negotiation failed: client connection to 127.3.12.65:35869: connect: Connection refused (error 111)
I20260805 23:07:54.061264 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 24485
I20260805 23:07:54.073259 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 23941
I20260805 23:07:54.084923 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37039
--webserver_interface=127.3.12.126
--webserver_port=41609
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:37039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:54.198987 24730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:54.199281 24730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:54.199357 24730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:54.203521 24730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:07:54.203617 24730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:54.203655 24730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:54.203694 24730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:07:54.203724 24730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:07:54.209131 24730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:37039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=41609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:54.210216 24730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:54.211248 24730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:54.220024 24741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:54.220147 24730 server_base.cc:1034] running on GCE node
W20260805 23:07:54.220057 24736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:54.220024 24735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:54.220942 24730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:54.222121 24730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:54.223330 24730 hybrid_clock.cc:648] HybridClock initialized: now 1785971274223310 us; error 36 us; skew 500 ppm
Aug 05 23:07:54 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971274, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:54.226470 24730 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:07:54.227428 24730 webserver.cc:460] Webserver started at http://127.3.12.126:41609/ using document root <none> and password file <none>
I20260805 23:07:54.228011 24730 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:54.228075 24730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:54.232390 24730 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:07:54.234611 24752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:54.236011 24730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:07:54.236160 24730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "628c3b4d034f45829c11dc189afb0163"
format_stamp: "Formatted at 2026-08-05 23:07:51 on dist-test-slave-2d19"
server_key: "e6e4d0ad104ff34ae4167fe41b914f81"
server_key_iv: "cec4e2cb02c22c4eb9de3407f402d6a5"
server_key_version: "encryptionkey@0"
I20260805 23:07:54.236827 24730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:54.253746 24730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:54.257354 24730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:54.257570 24730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:54.265055 24730 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:37039
I20260805 23:07:54.265363 24827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:37039 every 8 connection(s)
I20260805 23:07:54.266244 24730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:07:54.270403 24828 sys_catalog.cc:263] Verifying existing consensus state
I20260805 23:07:54.271982 24828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Bootstrap starting.
I20260805 23:07:54.272209 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 24730
I20260805 23:07:54.272663 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:35869
--local_ip_for_outbound_sockets=127.3.12.65
--tserver_master_addrs=127.3.12.126:37039
--webserver_port=46419
--webserver_interface=127.3.12.65
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:07:54.296247 24828 log.cc:824] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:54.310681 24828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:07:54.311131 24828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: Bootstrap complete.
I20260805 23:07:54.316689 24828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:54.317231 24828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628c3b4d034f45829c11dc189afb0163, State: Initialized, Role: FOLLOWER
I20260805 23:07:54.317836 24828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:54.318001 24828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:07:54.318077 24828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:07:54.318173 24828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 2 FOLLOWER]: Advancing to term 3
I20260805 23:07:54.320678 24828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:54.321040 24828 leader_election.cc:302] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 628c3b4d034f45829c11dc189afb0163; no voters:
I20260805 23:07:54.321381 24828 leader_election.cc:288] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260805 23:07:54.321520 24833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 23:07:54.321770 24833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [term 3 LEADER]: Becoming Leader. State: Replica: 628c3b4d034f45829c11dc189afb0163, State: Running, Role: LEADER
I20260805 23:07:54.322077 24833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } }
I20260805 23:07:54.322624 24828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:07:54.323629 24838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 628c3b4d034f45829c11dc189afb0163. Latest consensus state: current_term: 3 leader_uuid: "628c3b4d034f45829c11dc189afb0163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } } }
I20260805 23:07:54.323782 24838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:54.323618 24834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "628c3b4d034f45829c11dc189afb0163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628c3b4d034f45829c11dc189afb0163" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37039 } } }
I20260805 23:07:54.323879 24834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163 [sys.catalog]: This master's current role is: LEADER
I20260805 23:07:54.327919 24857 catalog_manager.cc:1259] Loaded cluster ID: 4a73ef5de62f4fc4b31cddca8ca60ec4
I20260805 23:07:54.327983 24857 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: loading cluster ID for follower catalog manager: success
W20260805 23:07:54.329176 24857 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 628c3b4d034f45829c11dc189afb0163: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260805 23:07:54.329252 24857 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260805 23:07:54.329339 24849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:07:54.332156 24849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:07:54.332720 24849 catalog_manager.cc:1259] Loaded cluster ID: 4a73ef5de62f4fc4b31cddca8ca60ec4
I20260805 23:07:54.332782 24849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260805 23:07:54.333459 24849 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff722f669ad google::LogMessage::Fail() at ??:0
@ 0x7ff722f6988a google::LogMessage::SendToLog() at ??:0
@ 0x7ff722f664de google::LogMessage::Flush() at ??:0
@ 0x7ff722f67a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff72d8a5fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff72d8a6d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff72d8a3579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff72d8e7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff7251b83d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff7251b8c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff7251ba3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff7251a9512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff7267ed6db start_thread at ??:0
@ 0x7ff721a6461f clone at ??:0
W20260805 23:07:54.417456 24830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:54.417739 24830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:54.417852 24830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:54.422012 24830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:54.422091 24830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:54.422201 24830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:07:54.427716 24830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:35869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=46419
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:54.428896 24830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:54.430001 24830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:07:54.439998 24830 server_base.cc:1034] running on GCE node
W20260805 23:07:54.439955 24890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:54.439927 24889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:54.439924 24892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:54.440702 24830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:54.441501 24830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:54.442711 24830 hybrid_clock.cc:648] HybridClock initialized: now 1785971274442670 us; error 48 us; skew 500 ppm
Aug 05 23:07:54 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971274, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:54.445999 24830 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:07:54.446938 24830 webserver.cc:460] Webserver started at http://127.3.12.65:46419/ using document root <none> and password file <none>
I20260805 23:07:54.447515 24830 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:54.447576 24830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:54.451551 24830 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:07:54.453871 24902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:54.455174 24830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:07:54.455317 24830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "764ee773f90e4b7a95d3b79d5bda74fb"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2fc060d10a33dde404787ac275f828f4"
server_key_iv: "5cdbb8dcacc37a76ca75734edf758567"
server_key_version: "encryptionkey@0"
I20260805 23:07:54.455899 24830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:54.474192 24830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:54.477695 24830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:54.477932 24830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:54.478610 24830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:54.479883 24911 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260805 23:07:54.485329 24830 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260805 23:07:54.485424 24830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260805 23:07:54.485530 24830 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260805 23:07:54.487315 24830 ts_tablet_manager.cc:610] Registered 1 tablets
I20260805 23:07:54.487388 24830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260805 23:07:54.487692 24911 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Bootstrap starting.
I20260805 23:07:54.498819 24830 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:35869
I20260805 23:07:54.498912 25066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:35869 every 8 connection(s)
I20260805 23:07:54.499845 24830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:07:54.501489 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 24830
I20260805 23:07:54.502029 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:45055
--local_ip_for_outbound_sockets=127.3.12.66
--tserver_master_addrs=127.3.12.126:37039
--webserver_port=37899
--webserver_interface=127.3.12.66
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:07:54.509030 24911 log.cc:824] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Log is configured to *not* fsync() on all Append() calls
I20260805 23:07:54.519620 24911 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:07:54.520090 24911 tablet_bootstrap.cc:492] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Bootstrap complete.
I20260805 23:07:54.521813 24911 ts_tablet_manager.cc:1397] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.008s
I20260805 23:07:54.525405 24911 raft_consensus.cc:348] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:54.526002 24911 raft_consensus.cc:729] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764ee773f90e4b7a95d3b79d5bda74fb, State: Initialized, Role: FOLLOWER
I20260805 23:07:54.526577 24911 consensus_queue.cc:260] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:54.527809 24911 ts_tablet_manager.cc:1428] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260805 23:07:54.629992 25075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:54.630399 25075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:54.630470 25075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:54.636528 25075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:54.636600 25075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:54.636704 25075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:07:54.642278 25075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:45055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=37899
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:54.643420 25075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:54.644556 25075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:54.654528 25087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:54.654556 25088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:54.654728 25090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:54.654989 25075 server_base.cc:1034] running on GCE node
I20260805 23:07:54.655403 25075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:54.656095 25075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:54.657311 25075 hybrid_clock.cc:648] HybridClock initialized: now 1785971274657280 us; error 49 us; skew 500 ppm
Aug 05 23:07:54 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971274, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:54.660703 25075 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:07:54.661921 25075 webserver.cc:460] Webserver started at http://127.3.12.66:37899/ using document root <none> and password file <none>
I20260805 23:07:54.662683 25075 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:54.662762 25075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:54.667627 25075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:07:54.670109 25101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:54.671446 25075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:07:54.671545 25075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "b955aa5503b947af9c2d77c1ade0c5a0"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "9e75a607616bd5a536ca0c2276742b28"
server_key_iv: "1e3ce28fc91bc2867adc16c59756a05a"
server_key_version: "encryptionkey@0"
I20260805 23:07:54.672104 25075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:54.707762 25075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:54.711786 25075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:54.712028 25075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:54.712860 25075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:54.713907 25075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:54.714000 25075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:54.714058 25075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:54.714092 25075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:54.724836 25075 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:45055
I20260805 23:07:54.724879 25267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:45055 every 8 connection(s)
I20260805 23:07:54.725893 25075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:07:54.734520 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 25075
I20260805 23:07:54.735011 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:43739
--local_ip_for_outbound_sockets=127.3.12.67
--tserver_master_addrs=127.3.12.126:37039
--webserver_port=35533
--webserver_interface=127.3.12.67
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:07:54.775075 25074 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:07:54.502134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:41411)
0805 23:07:54.504459 (+ 2325us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.504529 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:54.505631 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:54.774304 (+268673us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1965,"thread_start_us":192,"threads_started":1}
W20260805 23:07:54.775257 25271 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:07:54.728215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:46479)
0805 23:07:54.728787 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.728836 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:54.730205 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:54.774316 (+ 44111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":117,"threads_started":1}
W20260805 23:07:54.775835 25068 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:37039 (0 consecutive failures): Network error: Failed to ping master at 127.3.12.126:37039: Client connection negotiation failed: client connection to 127.3.12.126:37039: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260805 23:07:54.776068 25268 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:37039 (0 consecutive failures): Network error: Failed to ping master at 127.3.12.126:37039: Client connection negotiation failed: client connection to 127.3.12.126:37039: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260805 23:07:54.776664 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.776387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:50551)
0805 23:07:54.776513 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.776589 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260805 23:07:54.776875 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.776583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:59763)
0805 23:07:54.776752 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.776813 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260805 23:07:54.777495 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.777239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:59615)
0805 23:07:54.777368 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.777421 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260805 23:07:54.777570 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.777340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:59555)
0805 23:07:54.777443 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.777493 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260805 23:07:54.777705 25068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 23:07:54.777796 25268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 23:07:54.853063 25274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:07:54.853343 25274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:07:54.853418 25274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:07:54.857419 25274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:07:54.857499 25274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:07:54.857590 25274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:07:54.863016 25274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:43739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=35533
--tserver_master_addrs=127.3.12.126:37039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:07:54.864068 25274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:07:54.865140 25274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:07:54.874730 25283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:54.874730 25280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:07:54.874758 25285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:07:54.874967 25274 server_base.cc:1034] running on GCE node
I20260805 23:07:54.875607 25274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:07:54.876341 25274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:07:54.877552 25274 hybrid_clock.cc:648] HybridClock initialized: now 1785971274877538 us; error 29 us; skew 500 ppm
Aug 05 23:07:54 dist-test-slave-2d19 krb5kdc[23134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971274, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:07:54.880448 25274 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:07:54.881387 25274 webserver.cc:460] Webserver started at http://127.3.12.67:35533/ using document root <none> and password file <none>
I20260805 23:07:54.881980 25274 fs_manager.cc:359] Metadata directory not provided
I20260805 23:07:54.882112 25274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:07:54.886094 25274 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:07:54.888633 25297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:54.890033 25274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:07:54.890175 25274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "fe7a779e84604edb9972cdbc2a5aa932"
format_stamp: "Formatted at 2026-08-05 23:07:52 on dist-test-slave-2d19"
server_key: "2d9bbd0f9a6b5dfa446c82cba1fd7495"
server_key_iv: "5f6c198befceee7276e95eb36c4febe6"
server_key_version: "encryptionkey@0"
I20260805 23:07:54.890755 25274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:07:54.901855 25274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:07:54.905130 25274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:07:54.905360 25274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:07:54.906004 25274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:07:54.906863 25274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:07:54.906934 25274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:54.906997 25274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:07:54.907045 25274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:07:54.916656 25274 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:43739
I20260805 23:07:54.916683 25473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:43739 every 8 connection(s)
I20260805 23:07:54.917721 25274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:07:54.921712 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.919788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:37457)
0805 23:07:54.920344 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.920478 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
W20260805 23:07:54.922164 25474 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:37039 (0 consecutive failures): Network error: Failed to ping master at 127.3.12.126:37039: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
I20260805 23:07:54.922813 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.922581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:37689)
0805 23:07:54.922697 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.922750 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260805 23:07:54.922937 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 25274
I20260805 23:07:54.923686 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:54.923467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:33585)
0805 23:07:54.923582 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:54.923629 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260805 23:07:54.923883 25474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260805 23:07:55.778755 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:55.778305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:34995)
0805 23:07:55.778566 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:55.778635 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260805 23:07:55.778766 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:55.778348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:51597)
0805 23:07:55.778608 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:55.778661 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260805 23:07:55.924933 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:55.924529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:54965)
0805 23:07:55.924770 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:55.924826 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260805 23:07:56.309867 25493 raft_consensus.cc:482] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:07:56.310111 25493 raft_consensus.cc:504] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:56.311230 25493 leader_election.cc:288] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
Aug 05 23:07:56 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971274, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.67@KRBTEST.COM
Aug 05 23:07:56 dist-test-slave-2d19 krb5kdc[23134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971274, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.66@KRBTEST.COM
I20260805 23:07:56.326895 25501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:56.311700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:41837 (local address 127.3.12.66:45055)
0805 23:07:56.312133 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:56.312141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:56.312176 (+ 35us) server_negotiation.cc:407] Connection header received
0805 23:07:56.312394 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:56.312400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:56.312695 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:56.312913 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:56.314272 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:56.315834 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:56.317768 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:56.318493 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:56.318677 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:56.322292 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:56.322296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:56.322307 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:56.322336 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:56.324129 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:56.324839 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:56.324844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:56.324848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:56.324963 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:56.325334 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:56.325342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:56.325343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:56.325743 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:56.325892 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:56.326433 (+ 541us) server_negotiation.cc:299] Negotiation successful
0805 23:07:56.326687 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":103,"threads_started":1}
I20260805 23:07:56.326913 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:56.311483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:45055 (local address 127.3.12.65:41837)
0805 23:07:56.311818 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:56.311835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:56.312056 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:56.313058 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:56.313063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:56.313444 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:56.314067 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:56.314088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:56.316009 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:56.316013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:56.317633 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:56.317646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:56.318643 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:56.318647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:56.318986 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:56.319762 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:56.319791 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:56.322089 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:56.324282 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:56.324291 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:56.324307 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:56.324703 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:56.325069 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:56.325072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:56.325074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:56.325222 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:56.325867 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:56.325874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:56.326118 (+ 244us) client_negotiation.cc:769] Sending connection context
0805 23:07:56.326302 (+ 184us) client_negotiation.cc:240] Negotiation successful
0805 23:07:56.326672 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":317,"spinlock_wait_cycles":11904,"thread_start_us":66,"threads_started":1}
I20260805 23:07:56.326941 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:56.311697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:51405 (local address 127.3.12.67:43739)
0805 23:07:56.312137 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0805 23:07:56.312178 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0805 23:07:56.312215 (+ 37us) server_negotiation.cc:407] Connection header received
0805 23:07:56.313105 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:07:56.313113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:07:56.313508 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:07:56.313752 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:07:56.314272 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:56.315834 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:56.317768 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:07:56.318493 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:07:56.318667 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:56.322032 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:07:56.322037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:07:56.322050 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:07:56.322078 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:07:56.324164 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:56.324844 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:56.324852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:56.324862 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:56.325013 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:07:56.325348 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:07:56.325351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:07:56.325353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:07:56.325742 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:07:56.325899 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0805 23:07:56.326443 (+ 544us) server_negotiation.cc:299] Negotiation successful
0805 23:07:56.326740 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":113,"threads_started":1}
I20260805 23:07:56.327605 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:56.311483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43739 (local address 127.3.12.65:51405)
0805 23:07:56.311881 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:56.311896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:07:56.312056 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:07:56.313745 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:07:56.313748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:07:56.313932 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:07:56.314067 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:56.314088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:56.316009 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:56.316013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:56.317633 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:07:56.317646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:07:56.318643 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:07:56.318647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:07:56.318982 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:07:56.319762 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:07:56.319791 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:07:56.321766 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:07:56.324313 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:56.324318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:56.324320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:56.324719 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:56.325119 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:07:56.325123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:07:56.325125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:07:56.325241 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:07:56.325868 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:07:56.325875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:07:56.326118 (+ 243us) client_negotiation.cc:769] Sending connection context
0805 23:07:56.326302 (+ 184us) client_negotiation.cc:240] Negotiation successful
0805 23:07:56.326762 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":395,"spinlock_wait_cycles":70912,"thread_start_us":64,"threads_started":1}
W20260805 23:07:56.329658 24903 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
W20260805 23:07:56.329843 24907 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
I20260805 23:07:56.329923 24907 leader_election.cc:302] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb; no voters: b955aa5503b947af9c2d77c1ade0c5a0, fe7a779e84604edb9972cdbc2a5aa932
I20260805 23:07:56.330163 25493 raft_consensus.cc:2726] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:07:56.779891 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:56.779565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:38215)
0805 23:07:56.779713 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:56.779782 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260805 23:07:56.780144 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:56.779667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:57607)
0805 23:07:56.779929 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:56.780008 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260805 23:07:56.926163 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:56.925749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:47391)
0805 23:07:56.925999 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:56.926057 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260805 23:07:57.781219 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:57.780751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:36995)
0805 23:07:57.781030 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:57.781096 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260805 23:07:57.781319 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:57.780931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:44269)
0805 23:07:57.781176 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:57.781230 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260805 23:07:57.927376 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:57.926955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:56579)
0805 23:07:57.927206 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:57.927264 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260805 23:07:58.551961 25513 raft_consensus.cc:482] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:07:58.552152 25513 raft_consensus.cc:504] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:07:58.552711 25513 leader_election.cc:288] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
W20260805 23:07:58.553638 24903 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
W20260805 23:07:58.553733 24907 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
I20260805 23:07:58.553864 24907 leader_election.cc:302] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb; no voters: b955aa5503b947af9c2d77c1ade0c5a0, fe7a779e84604edb9972cdbc2a5aa932
I20260805 23:07:58.554008 25513 raft_consensus.cc:2726] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:07:58.782514 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:58.782070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:60597)
0805 23:07:58.782353 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:58.782412 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":105,"threads_started":1}
I20260805 23:07:58.782555 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:58.782120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:58273)
0805 23:07:58.782395 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:58.782449 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":114,"threads_started":1}
I20260805 23:07:58.928700 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:58.928176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:49703)
0805 23:07:58.928536 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:58.928600 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":153,"threads_started":1}
I20260805 23:07:59.783842 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:59.783398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:37827)
0805 23:07:59.783683 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:59.783740 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260805 23:07:59.783872 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:59.783442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:34477)
0805 23:07:59.783727 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:59.783786 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260805 23:07:59.930079 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:07:59.929568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:54931)
0805 23:07:59.929875 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0805 23:07:59.929939 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260805 23:08:00.727213 25522 raft_consensus.cc:482] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:08:00.727344 25522 raft_consensus.cc:504] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:08:00.727844 25522 leader_election.cc:288] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
W20260805 23:08:00.728679 24903 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
W20260805 23:08:00.728828 24907 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
I20260805 23:08:00.728916 24907 leader_election.cc:302] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb; no voters: b955aa5503b947af9c2d77c1ade0c5a0, fe7a779e84604edb9972cdbc2a5aa932
I20260805 23:08:00.729066 25522 raft_consensus.cc:2726] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:08:00.785118 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:00.784698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:44419)
0805 23:08:00.784941 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:00.785006 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":104,"threads_started":1}
I20260805 23:08:00.785161 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:00.784726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:55537)
0805 23:08:00.784994 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:00.785058 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":113,"threads_started":1}
I20260805 23:08:00.931345 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:00.930894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:56515)
0805 23:08:00.931180 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:00.931243 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":104,"threads_started":1}
I20260805 23:08:01.786578 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:01.786034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:35899)
0805 23:08:01.786349 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:01.786418 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
I20260805 23:08:01.787516 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:01.786893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:51757)
0805 23:08:01.787326 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:01.787412 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":70,"threads_started":1}
I20260805 23:08:01.932821 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:01.932236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:47031)
0805 23:08:01.932640 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:01.932717 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":77,"threads_started":1}
I20260805 23:08:02.595387 25535 raft_consensus.cc:482] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:08:02.595547 25535 raft_consensus.cc:504] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:08:02.595973 25535 leader_election.cc:288] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
W20260805 23:08:02.596879 24903 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
W20260805 23:08:02.597038 24907 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
I20260805 23:08:02.597174 24907 leader_election.cc:302] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb; no voters: b955aa5503b947af9c2d77c1ade0c5a0, fe7a779e84604edb9972cdbc2a5aa932
I20260805 23:08:02.597339 25535 raft_consensus.cc:2726] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:08:02.788029 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:02.787569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:50739)
0805 23:08:02.787860 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:02.787918 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
I20260805 23:08:02.789057 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:02.788663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:49477)
0805 23:08:02.788913 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:02.788975 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
I20260805 23:08:02.934139 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:02.933648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:60329)
0805 23:08:02.933944 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:02.934013 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260805 23:08:03.789569 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:03.788946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:55877)
0805 23:08:03.789319 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:03.789425 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20260805 23:08:03.790886 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:03.789871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:38583)
0805 23:08:03.790672 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:03.790759 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":82,"threads_started":1}
I20260805 23:08:03.935518 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:03.935008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:60155)
0805 23:08:03.935333 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:03.935398 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
I20260805 23:08:04.791441 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:04.790877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:45511)
0805 23:08:04.791219 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:04.791294 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":122,"threads_started":1}
I20260805 23:08:04.792316 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:04.791724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:57217)
0805 23:08:04.792095 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:04.792177 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":87,"threads_started":1}
I20260805 23:08:04.937006 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:04.936502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:41379)
0805 23:08:04.936820 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:04.936887 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":108,"threads_started":1}
I20260805 23:08:05.373780 25551 raft_consensus.cc:482] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:08:05.373978 25551 raft_consensus.cc:504] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:08:05.374872 25551 leader_election.cc:288] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
W20260805 23:08:05.375921 24903 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
W20260805 23:08:05.376085 24907 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
I20260805 23:08:05.376274 24907 leader_election.cc:302] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb; no voters: b955aa5503b947af9c2d77c1ade0c5a0, fe7a779e84604edb9972cdbc2a5aa932
I20260805 23:08:05.376502 25551 raft_consensus.cc:2726] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:08:05.793182 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:05.792532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:56883)
0805 23:08:05.792902 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:05.792970 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
I20260805 23:08:05.793653 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:05.793273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:49789)
0805 23:08:05.793519 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:05.793573 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20260805 23:08:05.938539 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:05.938036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:36747)
0805 23:08:05.938361 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:05.938432 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
I20260805 23:08:06.794941 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:06.794342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:58675)
0805 23:08:06.794741 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:06.794817 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":117,"threads_started":1}
I20260805 23:08:06.795033 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:06.794523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:38103)
0805 23:08:06.794817 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:06.794873 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":101,"threads_started":1}
I20260805 23:08:06.940094 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:06.939506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:34981)
0805 23:08:06.939849 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:06.939920 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260805 23:08:07.796403 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:07.795857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:49785)
0805 23:08:07.796185 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:07.796244 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":140,"threads_started":1}
I20260805 23:08:07.796458 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:07.795975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:42539)
0805 23:08:07.796282 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:07.796350 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":111,"threads_started":1}
I20260805 23:08:07.903053 25567 raft_consensus.cc:482] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:08:07.903201 25567 raft_consensus.cc:504] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7a779e84604edb9972cdbc2a5aa932" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43739 } } peers { permanent_uuid: "b955aa5503b947af9c2d77c1ade0c5a0" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 45055 } } peers { permanent_uuid: "764ee773f90e4b7a95d3b79d5bda74fb" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 35869 } }
I20260805 23:08:07.903725 25567 leader_election.cc:288] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739), b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055)
W20260805 23:08:07.904587 24903 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b955aa5503b947af9c2d77c1ade0c5a0 (127.3.12.66:45055): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
W20260805 23:08:07.904713 24907 leader_election.cc:341] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe7a779e84604edb9972cdbc2a5aa932 (127.3.12.67:43739): Not found: Tablet not found: 5f11ab6f09394fe3813ad3aadd999292
I20260805 23:08:07.904829 24907 leader_election.cc:302] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 764ee773f90e4b7a95d3b79d5bda74fb; no voters: b955aa5503b947af9c2d77c1ade0c5a0, fe7a779e84604edb9972cdbc2a5aa932
I20260805 23:08:07.905059 25567 raft_consensus.cc:2726] T 5f11ab6f09394fe3813ad3aadd999292 P 764ee773f90e4b7a95d3b79d5bda74fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:08:07.941690 25568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:07.941197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:60529)
0805 23:08:07.941499 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:07.941570 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
I20260805 23:08:08.797731 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:08.797252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:49041)
0805 23:08:08.797550 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:08.797627 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260805 23:08:08.797745 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:08.797316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:42639)
0805 23:08:08.797604 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:08.797660 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260805 23:08:08.943049 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:08.942580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:44777)
0805 23:08:08.942881 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:08.942940 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
I20260805 23:08:09.799229 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:09.798733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.66:55125)
0805 23:08:09.799066 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:09.799128 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20260805 23:08:09.799258 25575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:09.798776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.65:51715)
0805 23:08:09.799106 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:09.799171 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":62,"threads_started":1}
I20260805 23:08:09.925201 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 24830
I20260805 23:08:09.935869 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 25075
I20260805 23:08:09.945879 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:09.945108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37039 (local address 127.3.12.67:57015)
0805 23:08:09.945720 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:09.945780 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:37039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":283,"threads_started":1}
I20260805 23:08:09.950994 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 25274
2026-08-05T23:08:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19579 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:08:10 dist-test-slave-2d19 krb5kdc[25580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:08:10 dist-test-slave-2d19 krb5kdc[25580](info): set up 2 sockets
Aug 05 23:08:10 dist-test-slave-2d19 krb5kdc[25580](info): commencing operation
krb5kdc: starting...
W20260805 23:08:11.178493 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:08:11 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:08:11.311648 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40745
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:08:11.424495 25598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:11.424805 25598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:11.424885 25598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:11.429458 25598 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260805 23:08:11.429546 25598 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260805 23:08:11.429574 25598 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260805 23:08:11.429605 25598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:08:11.429668 25598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:11.429709 25598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:11.429762 25598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:08:11.429800 25598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:08:11.439119 25598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:11.440565 25598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:11.441906 25598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:11.450879 25604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:11.450920 25598 server_base.cc:1034] running on GCE node
W20260805 23:08:11.450940 25603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:11.450882 25606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:11.451915 25598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:11.453030 25598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:11.454206 25598 hybrid_clock.cc:648] HybridClock initialized: now 1785971291454187 us; error 33 us; skew 500 ppm
Aug 05 23:08:11 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971291, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:11.457186 25598 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:08:11.458091 25598 webserver.cc:460] Webserver started at http://127.3.12.126:46861/ using document root <none> and password file <none>
I20260805 23:08:11.458652 25598 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:11.458745 25598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:11.458986 25598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:08:11.461061 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "d1a6e904d8bd46da805ae38dace975ad"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "d3342164c422ca7c877470c5f44685f0"
server_key_iv: "c62fafbd2f2f407bea76308ec6381b58"
server_key_version: "encryptionkey@0"
I20260805 23:08:11.461629 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "d1a6e904d8bd46da805ae38dace975ad"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "d3342164c422ca7c877470c5f44685f0"
server_key_iv: "c62fafbd2f2f407bea76308ec6381b58"
server_key_version: "encryptionkey@0"
I20260805 23:08:11.465305 25598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:08:11.467633 25615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:11.468847 25598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:08:11.469018 25598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "d1a6e904d8bd46da805ae38dace975ad"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "d3342164c422ca7c877470c5f44685f0"
server_key_iv: "c62fafbd2f2f407bea76308ec6381b58"
server_key_version: "encryptionkey@0"
I20260805 23:08:11.469133 25598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:11.499127 25598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:11.502178 25598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:11.502396 25598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:11.510095 25598 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:40745
I20260805 23:08:11.510403 25690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:40745 every 8 connection(s)
I20260805 23:08:11.511099 25598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:08:11.513964 25693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:08:11.519418 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 25598
I20260805 23:08:11.519558 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:08:11.519876 3121 external_mini_cluster.cc:1428] Setting key f91e0b4eee08e056ad5e5aefde6cafda
I20260805 23:08:11.520107 25693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Bootstrap starting.
I20260805 23:08:11.522812 25693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Neither blocks nor log segments found. Creating new log.
I20260805 23:08:11.523834 25693 log.cc:824] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:11.526131 25693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: No bootstrap required, opened a new log
Aug 05 23:08:11 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:08:11.530889 25693 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:11.531150 25693 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:08:11.531234 25693 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a6e904d8bd46da805ae38dace975ad, State: Initialized, Role: FOLLOWER
I20260805 23:08:11.531790 25693 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:11.531978 25693 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:08:11.532052 25693 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:08:11.532187 25693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:08:11.533164 25693 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:11.533516 25693 leader_election.cc:302] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a6e904d8bd46da805ae38dace975ad; no voters:
I20260805 23:08:11.533879 25693 leader_election.cc:288] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:08:11.533957 25702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:08:11.534165 25702 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 1 LEADER]: Becoming Leader. State: Replica: d1a6e904d8bd46da805ae38dace975ad, State: Running, Role: LEADER
I20260805 23:08:11.534473 25702 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:11.535243 25693 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:08:11.535669 25702 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a6e904d8bd46da805ae38dace975ad. Latest consensus state: current_term: 1 leader_uuid: "d1a6e904d8bd46da805ae38dace975ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } } }
I20260805 23:08:11.535786 25702 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: This master's current role is: LEADER
I20260805 23:08:11.536168 25706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:08:11.536520 25703 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1a6e904d8bd46da805ae38dace975ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } } }
I20260805 23:08:11.536633 25703 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: This master's current role is: LEADER
I20260805 23:08:11.536648 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:11.521369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.3.12.126:40745)
0805 23:08:11.521804 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:11.521824 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:11.521863 (+ 39us) server_negotiation.cc:407] Connection header received
0805 23:08:11.522531 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:11.522549 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:11.522916 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:11.523234 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:11.524216 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:11.525442 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:11.526800 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:11.527759 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:11.527989 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:11.531000 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:11.531005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:11.531020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:11.531049 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:11.533001 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:11.533543 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:11.533551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:11.533562 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:11.533735 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:11.534094 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:11.534097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:11.534099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:11.534707 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:11.534906 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:11.535229 (+ 323us) server_negotiation.cc:299] Negotiation successful
0805 23:08:11.535557 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20260805 23:08:11.539826 25706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:08:11.545759 25706 catalog_manager.cc:1347] Generated new cluster ID: 087fe85f9a1845fc964846c0bfcaa416
I20260805 23:08:11.545846 25706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:08:11.566958 25706 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:08:11.567934 25706 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:08:11.575477 25706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Generated new TSK 0
I20260805 23:08:11.576438 25706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:08:11.615386 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40745
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:08:11.729859 25731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:11.730141 25731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:11.730185 25731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:11.734457 25731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:11.734529 25731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:11.734618 25731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:08:11.740224 25731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:11.741454 25731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:11.742511 25731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:11.752187 25742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:11.752194 25737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:11.752187 25739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:11.752481 25731 server_base.cc:1034] running on GCE node
I20260805 23:08:11.752982 25731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:11.753631 25731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:11.754794 25731 hybrid_clock.cc:648] HybridClock initialized: now 1785971291754770 us; error 41 us; skew 500 ppm
Aug 05 23:08:11 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971291, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:11.757753 25731 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:08:11.758662 25731 webserver.cc:460] Webserver started at http://127.3.12.65:37455/ using document root <none> and password file <none>
I20260805 23:08:11.759189 25731 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:11.759248 25731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:11.759409 25731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:08:11.761497 25731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "00d3a98a48cea03dcab112078bd5107b"
server_key_iv: "54a6e6c91c5cdbcb1c0bed33867d0f7d"
server_key_version: "encryptionkey@0"
I20260805 23:08:11.762063 25731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "00d3a98a48cea03dcab112078bd5107b"
server_key_iv: "54a6e6c91c5cdbcb1c0bed33867d0f7d"
server_key_version: "encryptionkey@0"
I20260805 23:08:11.765736 25731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:08:11.768129 25752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:11.769436 25731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:08:11.769577 25731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "00d3a98a48cea03dcab112078bd5107b"
server_key_iv: "54a6e6c91c5cdbcb1c0bed33867d0f7d"
server_key_version: "encryptionkey@0"
I20260805 23:08:11.769698 25731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:11.788233 25731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:11.791772 25731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:11.792011 25731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:11.792692 25731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:11.793565 25731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:08:11.793637 25731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:11.793710 25731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:08:11.793763 25731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:11.803182 25731 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:41295
I20260805 23:08:11.803210 25916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:41295 every 8 connection(s)
I20260805 23:08:11.804124 25731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:08:11.812604 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 25731
I20260805 23:08:11.812783 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:08:11.813163 3121 external_mini_cluster.cc:1428] Setting key 2af983a062e48a17e09b382da1ff3a51
Aug 05 23:08:11 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971291, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:08:11.819048 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:11.806132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:47283 (local address 127.3.12.126:40745)
0805 23:08:11.806269 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:11.806273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:11.807092 (+ 819us) server_negotiation.cc:407] Connection header received
0805 23:08:11.808508 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:11.808511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:11.808572 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:11.808709 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:11.809884 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:11.810450 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:11.811587 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:11.811821 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:11.811958 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:11.814876 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:11.814880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:11.814882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:11.814904 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:11.816487 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:11.817117 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:11.817122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:11.817125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:11.817227 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:11.817662 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:11.817667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:11.817670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:11.817916 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:11.818073 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:11.818750 (+ 677us) server_negotiation.cc:299] Negotiation successful
0805 23:08:11.818880 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 23:08:11.820055 25921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:11.806444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:47283)
0805 23:08:11.806923 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:11.806978 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:11.808213 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:11.808882 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:11.808889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:11.809262 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:11.809736 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:11.809748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:11.810583 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:11.810587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:11.811459 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:11.811468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:11.812030 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:11.812036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:11.812197 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:11.813042 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:11.813064 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:11.814707 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:11.816601 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:11.816610 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:11.816629 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:11.817002 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:11.817329 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:11.817331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:11.817332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:11.817507 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:11.818073 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:11.818080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:11.818483 (+ 403us) client_negotiation.cc:769] Sending connection context
0805 23:08:11.818707 (+ 224us) client_negotiation.cc:240] Negotiation successful
0805 23:08:11.819054 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260805 23:08:11.821771 25917 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40745
I20260805 23:08:11.822137 25917 heartbeater.cc:464] Registering TS with master...
I20260805 23:08:11.822813 25917 heartbeater.cc:511] Master 127.3.12.126:40745 requested a full tablet report, sending...
I20260805 23:08:11.824579 25635 ts_manager.cc:194] Registered new tserver with Master: 742a0c3bcf0249fda5da0ab1d4bc2b37 (127.3.12.65:41295)
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:08:11.825812 25635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:47283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:08:11.847456 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40745
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:08:11.963582 25925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:11.963871 25925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:11.964011 25925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:11.968453 25925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:11.968538 25925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:11.968626 25925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:08:11.974275 25925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:11.975370 25925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:11.976497 25925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:11.986337 25932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:11.986361 25931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:11.986421 25925 server_base.cc:1034] running on GCE node
W20260805 23:08:11.986361 25935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:11.987093 25925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:11.987736 25925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:11.988903 25925 hybrid_clock.cc:648] HybridClock initialized: now 1785971291988886 us; error 30 us; skew 500 ppm
Aug 05 23:08:11 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971291, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:11.991909 25925 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:08:11.992937 25925 webserver.cc:460] Webserver started at http://127.3.12.66:38103/ using document root <none> and password file <none>
I20260805 23:08:11.993532 25925 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:11.993630 25925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:11.993870 25925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:08:11.995985 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "2542b95e31e7482690c3246f515721fc"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "e0075bee5a2813662e67096a13c1b52b"
server_key_iv: "22e02e26c62ea3179f3dda05ac312b6e"
server_key_version: "encryptionkey@0"
I20260805 23:08:11.996580 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "2542b95e31e7482690c3246f515721fc"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "e0075bee5a2813662e67096a13c1b52b"
server_key_iv: "22e02e26c62ea3179f3dda05ac312b6e"
server_key_version: "encryptionkey@0"
I20260805 23:08:12.000509 25925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260805 23:08:12.003125 25947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:12.004557 25925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:08:12.004698 25925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "2542b95e31e7482690c3246f515721fc"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "e0075bee5a2813662e67096a13c1b52b"
server_key_iv: "22e02e26c62ea3179f3dda05ac312b6e"
server_key_version: "encryptionkey@0"
I20260805 23:08:12.004817 25925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:12.019346 25925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:12.022821 25925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:12.023053 25925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:12.023687 25925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:12.024757 25925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:08:12.024854 25925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:12.024919 25925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:08:12.024948 25925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:12.034487 25925 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:33093
I20260805 23:08:12.034509 26120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:33093 every 8 connection(s)
I20260805 23:08:12.035616 25925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:08:12.045316 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 25925
I20260805 23:08:12.045452 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:08:12.045758 3121 external_mini_cluster.cc:1428] Setting key ca2d71c47002394c044d234039eb9f01
Aug 05 23:08:12 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971291, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:08:12.052135 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.038007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:37753 (local address 127.3.12.126:40745)
0805 23:08:12.038175 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.038179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.039091 (+ 912us) server_negotiation.cc:407] Connection header received
0805 23:08:12.040678 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.040681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.040746 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.040867 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:12.042176 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.042773 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.044039 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.044366 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.044500 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.047620 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:12.047624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:12.047628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:12.047658 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:12.049510 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:12.050244 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:12.050247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:12.050249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:12.050326 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:12.050787 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:12.050790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:12.050791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:12.050990 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:12.051072 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.051728 (+ 656us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.051876 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 23:08:12.053014 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.038424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:37753)
0805 23:08:12.038926 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:12.038974 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:12.040446 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:12.041072 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:12.041080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:12.041474 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:12.042007 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.042020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.042931 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.042935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.043884 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.043894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.044559 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.044564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.044743 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.045609 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:12.045628 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:12.047416 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:12.049658 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:12.049668 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:12.049690 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:12.050131 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:12.050428 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:12.050431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:12.050433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:12.050669 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:12.051109 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:12.051118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:12.051457 (+ 339us) client_negotiation.cc:769] Sending connection context
0805 23:08:12.051640 (+ 183us) client_negotiation.cc:240] Negotiation successful
0805 23:08:12.051963 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260805 23:08:12.054425 26123 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40745
I20260805 23:08:12.054705 26123 heartbeater.cc:464] Registering TS with master...
I20260805 23:08:12.055246 26123 heartbeater.cc:511] Master 127.3.12.126:40745 requested a full tablet report, sending...
I20260805 23:08:12.056488 25635 ts_manager.cc:194] Registered new tserver with Master: 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093)
I20260805 23:08:12.057322 25635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:37753
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:08:12.077615 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40745
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:08:12.192622 26128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:12.192888 26128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:12.192934 26128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:12.197026 26128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:12.197096 26128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:12.197156 26128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:08:12.202381 26128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:12.203406 26128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:12.204480 26128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:12.214386 26133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:12.214435 26134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:12.214557 26128 server_base.cc:1034] running on GCE node
W20260805 23:08:12.214573 26138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:12.215134 26128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:12.215718 26128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:12.216881 26128 hybrid_clock.cc:648] HybridClock initialized: now 1785971292216858 us; error 35 us; skew 500 ppm
Aug 05 23:08:12 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971292, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:12.219702 26128 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:08:12.220723 26128 webserver.cc:460] Webserver started at http://127.3.12.67:46827/ using document root <none> and password file <none>
I20260805 23:08:12.221306 26128 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:12.221369 26128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:12.221597 26128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:08:12.223707 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "d4184e56e8404a4e9a34becca764ec9e"
format_stamp: "Formatted at 2026-08-05 23:08:12 on dist-test-slave-2d19"
server_key: "72fa3efb922d0789ab53109344e82f12"
server_key_iv: "814df1abde6b32c2b92a6867ee128ac3"
server_key_version: "encryptionkey@0"
I20260805 23:08:12.224412 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "d4184e56e8404a4e9a34becca764ec9e"
format_stamp: "Formatted at 2026-08-05 23:08:12 on dist-test-slave-2d19"
server_key: "72fa3efb922d0789ab53109344e82f12"
server_key_iv: "814df1abde6b32c2b92a6867ee128ac3"
server_key_version: "encryptionkey@0"
I20260805 23:08:12.228497 26128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:08:12.231205 26149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:12.232633 26128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:08:12.232779 26128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "d4184e56e8404a4e9a34becca764ec9e"
format_stamp: "Formatted at 2026-08-05 23:08:12 on dist-test-slave-2d19"
server_key: "72fa3efb922d0789ab53109344e82f12"
server_key_iv: "814df1abde6b32c2b92a6867ee128ac3"
server_key_version: "encryptionkey@0"
I20260805 23:08:12.232904 26128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:12.251295 26128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:12.254594 26128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:12.254827 26128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:12.255487 26128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:12.256417 26128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:08:12.256487 26128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:12.256553 26128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:08:12.256611 26128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:12.266227 26128 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:44123
I20260805 23:08:12.266258 26341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:44123 every 8 connection(s)
I20260805 23:08:12.267144 26128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:08:12.275357 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 26128
I20260805 23:08:12.275521 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:08:12.275812 3121 external_mini_cluster.cc:1428] Setting key 58d014d1b8072da381793ab96ec20538
Aug 05 23:08:12 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971292, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:08:12.281095 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.269118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:58665 (local address 127.3.12.126:40745)
0805 23:08:12.269305 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.269310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.270129 (+ 819us) server_negotiation.cc:407] Connection header received
0805 23:08:12.271373 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.271376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.271423 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.271498 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:12.272802 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.273392 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.274657 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.274882 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.274994 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.277611 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:12.277615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:12.277618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:12.277646 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:12.279207 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:12.279699 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:12.279702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:12.279703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:12.279781 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:12.280089 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:12.280091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:12.280092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:12.280322 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:12.280400 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.280866 (+ 466us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.280965 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260805 23:08:12.281881 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.269502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:58665)
0805 23:08:12.269972 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:12.270018 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:12.271184 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:12.271659 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:12.271665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:12.272036 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:12.272671 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.272687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.273529 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.273534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.274528 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.274539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.275049 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.275054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.275233 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.275938 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:12.275963 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:12.277454 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:12.279318 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:12.279323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:12.279336 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:12.279606 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:12.279863 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:12.279865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:12.279867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:12.280007 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:12.280403 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:12.280408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:12.280646 (+ 238us) client_negotiation.cc:769] Sending connection context
0805 23:08:12.280805 (+ 159us) client_negotiation.cc:240] Negotiation successful
0805 23:08:12.281065 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":121,"threads_started":1}
I20260805 23:08:12.282996 26342 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40745
I20260805 23:08:12.283249 26342 heartbeater.cc:464] Registering TS with master...
I20260805 23:08:12.283754 26342 heartbeater.cc:511] Master 127.3.12.126:40745 requested a full tablet report, sending...
I20260805 23:08:12.284834 25635 ts_manager.cc:194] Registered new tserver with Master: d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123)
I20260805 23:08:12.285396 25635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:58665
I20260805 23:08:12.289968 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:08:12.299755 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.292040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.3.12.126:40745)
0805 23:08:12.292236 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.292243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.292456 (+ 213us) server_negotiation.cc:407] Connection header received
0805 23:08:12.292608 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.292611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.292661 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.292749 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:12.293493 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.294005 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.294871 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.295115 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.295227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.296712 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:12.296716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:12.296719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:12.296744 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:12.298006 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:12.298445 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:12.298448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:12.298450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:12.298545 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:12.298830 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:12.298833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:12.298835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:12.299060 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:12.299205 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.299443 (+ 238us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.299589 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260805 23:08:12.302829 25635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:08:12.306056 25635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:08:12.321688 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.315370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.3.12.66:33093)
0805 23:08:12.315909 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.315924 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.316245 (+ 321us) server_negotiation.cc:407] Connection header received
0805 23:08:12.316501 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.316509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.316736 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.316911 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:12.317224 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.318474 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.320324 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.321005 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.321123 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.321279 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.321431 (+ 152us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.321531 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":181,"threads_started":1}
I20260805 23:08:12.321674 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.315198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:33093 (local address 127.0.0.1:49420)
0805 23:08:12.316100 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:12.316152 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:12.316338 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:12.316909 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:12.316917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:12.316941 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:12.317101 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.317108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.318593 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.318596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.320177 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.320184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.321134 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.321138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.321240 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.321280 (+ 40us) client_negotiation.cc:769] Sending connection context
0805 23:08:12.321394 (+ 114us) client_negotiation.cc:240] Negotiation successful
0805 23:08:12.321525 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":175,"thread_start_us":80,"threads_started":1}
I20260805 23:08:12.322021 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.316412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.3.12.67:44123)
0805 23:08:12.316899 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.316911 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.316931 (+ 20us) server_negotiation.cc:407] Connection header received
0805 23:08:12.317001 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.317007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.317204 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.317360 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:12.317591 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.318377 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.320336 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.321082 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.321185 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.321321 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.321785 (+ 464us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.321871 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":161,"threads_started":1}
I20260805 23:08:12.321905 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.315698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:41295 (local address 127.0.0.1:48174)
0805 23:08:12.316105 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:12.316153 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:12.316352 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:12.317538 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:12.317542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:12.317560 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:12.317639 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.317645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.319028 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.319033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.320048 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.320063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.321390 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.321394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.321549 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.321594 (+ 45us) client_negotiation.cc:769] Sending connection context
0805 23:08:12.321667 (+ 73us) client_negotiation.cc:240] Negotiation successful
0805 23:08:12.321770 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":55,"thread_start_us":96,"threads_started":1}
I20260805 23:08:12.322257 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.316214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48174 (local address 127.3.12.65:41295)
0805 23:08:12.316807 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.316823 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.316847 (+ 24us) server_negotiation.cc:407] Connection header received
0805 23:08:12.317189 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.317195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.317384 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.317549 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:12.317757 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.318869 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.320189 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.321076 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.321185 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.321321 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.322019 (+ 698us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.322126 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":126,"threads_started":1}
I20260805 23:08:12.322322 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.315959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:44123 (local address 127.0.0.1:51344)
0805 23:08:12.316596 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:12.316610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:12.316690 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:12.317349 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:12.317352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:12.317369 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:12.317466 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.317472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.318521 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.318525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.320176 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.320187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.321390 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.321394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.321562 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.321595 (+ 33us) client_negotiation.cc:769] Sending connection context
0805 23:08:12.321667 (+ 72us) client_negotiation.cc:240] Negotiation successful
0805 23:08:12.322197 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":705,"thread_start_us":87,"threads_started":1}
I20260805 23:08:12.324966 26240 tablet_service.cc:1467] Processing CreateTablet for tablet bb75f9508e344116bc17d7678161de32 (DEFAULT_TABLE table=test-table [id=e02d0fd59d8045c486eb2156edf127b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:08:12.325054 26022 tablet_service.cc:1467] Processing CreateTablet for tablet bb75f9508e344116bc17d7678161de32 (DEFAULT_TABLE table=test-table [id=e02d0fd59d8045c486eb2156edf127b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:08:12.325052 25820 tablet_service.cc:1467] Processing CreateTablet for tablet bb75f9508e344116bc17d7678161de32 (DEFAULT_TABLE table=test-table [id=e02d0fd59d8045c486eb2156edf127b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:08:12.325868 26240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb75f9508e344116bc17d7678161de32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:08:12.326048 25820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb75f9508e344116bc17d7678161de32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:08:12.326048 26022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb75f9508e344116bc17d7678161de32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:08:12.332294 26373 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: Bootstrap starting.
I20260805 23:08:12.333623 26375 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: Bootstrap starting.
I20260805 23:08:12.334519 26373 tablet_bootstrap.cc:654] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: Neither blocks nor log segments found. Creating new log.
I20260805 23:08:12.335656 26373 log.cc:824] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:12.335688 26375 tablet_bootstrap.cc:654] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: Neither blocks nor log segments found. Creating new log.
I20260805 23:08:12.336413 26374 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Bootstrap starting.
I20260805 23:08:12.336597 26375 log.cc:824] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:12.338461 26375 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: No bootstrap required, opened a new log
I20260805 23:08:12.338461 26373 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: No bootstrap required, opened a new log
I20260805 23:08:12.338495 26374 tablet_bootstrap.cc:654] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Neither blocks nor log segments found. Creating new log.
I20260805 23:08:12.338660 26373 ts_tablet_manager.cc:1397] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260805 23:08:12.338660 26375 ts_tablet_manager.cc:1397] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 23:08:12.339218 26374 log.cc:824] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:12.340973 26374 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: No bootstrap required, opened a new log
I20260805 23:08:12.341174 26374 ts_tablet_manager.cc:1397] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 23:08:12.342584 26373 raft_consensus.cc:348] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.342584 26375 raft_consensus.cc:348] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.342870 26375 raft_consensus.cc:374] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:08:12.342870 26373 raft_consensus.cc:374] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:08:12.342941 26373 raft_consensus.cc:729] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4184e56e8404a4e9a34becca764ec9e, State: Initialized, Role: FOLLOWER
I20260805 23:08:12.342932 26375 raft_consensus.cc:729] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2542b95e31e7482690c3246f515721fc, State: Initialized, Role: FOLLOWER
I20260805 23:08:12.343407 26375 consensus_queue.cc:260] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.343407 26373 consensus_queue.cc:260] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.344465 26375 ts_tablet_manager.cc:1428] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 23:08:12.344686 26373 ts_tablet_manager.cc:1428] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 23:08:12.344866 26123 heartbeater.cc:503] Master 127.3.12.126:40745 was elected leader, sending a full tablet report...
I20260805 23:08:12.344990 26342 heartbeater.cc:503] Master 127.3.12.126:40745 was elected leader, sending a full tablet report...
I20260805 23:08:12.345122 26374 raft_consensus.cc:348] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.345357 26374 raft_consensus.cc:374] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:08:12.345403 26374 raft_consensus.cc:729] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742a0c3bcf0249fda5da0ab1d4bc2b37, State: Initialized, Role: FOLLOWER
I20260805 23:08:12.345821 26374 consensus_queue.cc:260] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.346477 25917 heartbeater.cc:503] Master 127.3.12.126:40745 was elected leader, sending a full tablet report...
I20260805 23:08:12.346825 26374 ts_tablet_manager.cc:1428] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 23:08:12.390367 26388 raft_consensus.cc:482] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:08:12.390674 26388 raft_consensus.cc:504] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.392307 26388 leader_election.cc:288] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093), d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123)
I20260805 23:08:12.396505 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.392737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:46605 (local address 127.3.12.66:33093)
0805 23:08:12.392913 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.392919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.392996 (+ 77us) server_negotiation.cc:407] Connection header received
0805 23:08:12.393205 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.393209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.393270 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.393320 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:12.393784 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.394291 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.395493 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.395890 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.396017 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.396081 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.396281 (+ 200us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.396365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260805 23:08:12.396504 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.393123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:56193 (local address 127.3.12.67:44123)
0805 23:08:12.393248 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.393254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.393337 (+ 83us) server_negotiation.cc:407] Connection header received
0805 23:08:12.393471 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.393475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.393532 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.393615 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:12.393863 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.394356 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.395493 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.395907 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.396029 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.396092 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.396294 (+ 202us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.396364 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:08:12.396536 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.392516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:33093 (local address 127.3.12.65:46605)
0805 23:08:12.392882 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:12.392895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:12.393054 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:12.393501 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:12.393504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:12.393534 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:12.393683 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.393693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.394412 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.394416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.395355 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.395365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.396024 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.396027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.396122 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.396146 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 23:08:12.396245 (+ 99us) client_negotiation.cc:240] Negotiation successful
0805 23:08:12.396341 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":39,"thread_start_us":152,"threads_started":1}
I20260805 23:08:12.396543 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.392962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:44123 (local address 127.3.12.65:56193)
0805 23:08:12.393270 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:12.393282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:12.393367 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:12.393619 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:12.393621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:12.393633 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:12.393742 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.393746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.394470 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.394474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.395353 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:12.395363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.396034 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.396038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:12.396154 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.396172 (+ 18us) client_negotiation.cc:769] Sending connection context
0805 23:08:12.396257 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 23:08:12.396358 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":41,"thread_start_us":116,"threads_started":1}
I20260805 23:08:12.397181 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb75f9508e344116bc17d7678161de32" candidate_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2542b95e31e7482690c3246f515721fc" is_pre_election: true
I20260805 23:08:12.397415 26273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb75f9508e344116bc17d7678161de32" candidate_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4184e56e8404a4e9a34becca764ec9e" is_pre_election: true
I20260805 23:08:12.397505 26046 raft_consensus.cc:2454] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 742a0c3bcf0249fda5da0ab1d4bc2b37 in term 0.
I20260805 23:08:12.397711 26273 raft_consensus.cc:2454] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 742a0c3bcf0249fda5da0ab1d4bc2b37 in term 0.
I20260805 23:08:12.398025 25753 leader_election.cc:302] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2542b95e31e7482690c3246f515721fc, 742a0c3bcf0249fda5da0ab1d4bc2b37; no voters:
I20260805 23:08:12.398288 26388 raft_consensus.cc:2781] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:08:12.398379 26388 raft_consensus.cc:482] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:08:12.398425 26388 raft_consensus.cc:3037] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:08:12.399479 26388 raft_consensus.cc:504] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.399822 26388 leader_election.cc:288] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 1 election: Requested vote from peers 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093), d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123)
I20260805 23:08:12.400223 26046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb75f9508e344116bc17d7678161de32" candidate_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2542b95e31e7482690c3246f515721fc"
I20260805 23:08:12.400317 26273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb75f9508e344116bc17d7678161de32" candidate_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4184e56e8404a4e9a34becca764ec9e"
I20260805 23:08:12.400393 26046 raft_consensus.cc:3037] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:08:12.400460 26273 raft_consensus.cc:3037] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:08:12.401396 26046 raft_consensus.cc:2454] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 742a0c3bcf0249fda5da0ab1d4bc2b37 in term 1.
I20260805 23:08:12.401603 26273 raft_consensus.cc:2454] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 742a0c3bcf0249fda5da0ab1d4bc2b37 in term 1.
I20260805 23:08:12.401753 25753 leader_election.cc:302] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2542b95e31e7482690c3246f515721fc, 742a0c3bcf0249fda5da0ab1d4bc2b37; no voters:
I20260805 23:08:12.401940 26388 raft_consensus.cc:2781] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:08:12.402206 26388 raft_consensus.cc:686] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 LEADER]: Becoming Leader. State: Replica: 742a0c3bcf0249fda5da0ab1d4bc2b37, State: Running, Role: LEADER
I20260805 23:08:12.402566 26388 consensus_queue.cc:237] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:12.406052 25634 catalog_manager.cc:5495] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 742a0c3bcf0249fda5da0ab1d4bc2b37 (127.3.12.65). New cstate: current_term: 1 leader_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } health_report { overall_health: HEALTHY } } }
I20260805 23:08:12.419026 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.415527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.3.12.65:41295)
0805 23:08:12.415658 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:12.415661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:12.415675 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:08:12.415812 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:12.415814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:12.415867 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:12.415951 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:08:12.416206 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.416732 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.417643 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:12.417844 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:12.417919 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:12.418232 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:08:12.418620 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:08:12.418707 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:12.418834 (+ 127us) server_negotiation.cc:299] Negotiation successful
0805 23:08:12.418884 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:08:12.425182 26046 raft_consensus.cc:1261] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 1 FOLLOWER]: Refusing update from remote peer 742a0c3bcf0249fda5da0ab1d4bc2b37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:08:12.425182 26273 raft_consensus.cc:1261] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 1 FOLLOWER]: Refusing update from remote peer 742a0c3bcf0249fda5da0ab1d4bc2b37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:08:12.425940 26388 consensus_queue.cc:1035] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:08:12.426159 26394 consensus_queue.cc:1035] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:08:12.435812 26401 mvcc.cc:205] Tried to move back new op lower bound from 7315338413765672960 to 7315338413683601408. Current Snapshot: MvccSnapshot[applied={T|T < 7315338413765672960}]
I20260805 23:08:12.436028 26400 mvcc.cc:205] Tried to move back new op lower bound from 7315338413765672960 to 7315338413683601408. Current Snapshot: MvccSnapshot[applied={T|T < 7315338413765672960}]
I20260805 23:08:12.442925 25635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:08:12.443127 25635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:08:12.445928 25635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Sending DeleteTablet for 3 replicas of tablet bb75f9508e344116bc17d7678161de32
I20260805 23:08:12.446807 25820 tablet_service.cc:1514] Processing DeleteTablet for tablet bb75f9508e344116bc17d7678161de32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:08:12 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:48174
I20260805 23:08:12.446864 26022 tablet_service.cc:1514] Processing DeleteTablet for tablet bb75f9508e344116bc17d7678161de32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:08:12 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:49420
I20260805 23:08:12.447394 26240 tablet_service.cc:1514] Processing DeleteTablet for tablet bb75f9508e344116bc17d7678161de32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:08:12 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:51344
I20260805 23:08:12.447893 26417 tablet_replica.cc:330] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: stopping tablet replica
I20260805 23:08:12.447962 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 25731
I20260805 23:08:12.457871 26416 tablet_replica.cc:330] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: stopping tablet replica
I20260805 23:08:12.458276 26416 raft_consensus.cc:2229] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:08:12.458640 26416 raft_consensus.cc:2258] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:08:12.461313 26417 raft_consensus.cc:2229] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:08:12.461694 26417 raft_consensus.cc:2258] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:08:12.462585 26416 ts_tablet_manager.cc:1905] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:08:12.463243 26417 ts_tablet_manager.cc:1905] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:08:12.465570 26416 ts_tablet_manager.cc:1918] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:08:12.465679 26416 log.cc:1196] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/bb75f9508e344116bc17d7678161de32
I20260805 23:08:12.466075 26416 ts_tablet_manager.cc:1939] T bb75f9508e344116bc17d7678161de32 P 2542b95e31e7482690c3246f515721fc: Deleting consensus metadata
I20260805 23:08:12.466091 26417 ts_tablet_manager.cc:1918] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:08:12.466174 26417 log.cc:1196] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/bb75f9508e344116bc17d7678161de32
I20260805 23:08:12.466521 26417 ts_tablet_manager.cc:1939] T bb75f9508e344116bc17d7678161de32 P d4184e56e8404a4e9a34becca764ec9e: Deleting consensus metadata
I20260805 23:08:12.467499 25616 catalog_manager.cc:4841] TS 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093): tablet bb75f9508e344116bc17d7678161de32 (table test-table [id=e02d0fd59d8045c486eb2156edf127b0]) successfully deleted
W20260805 23:08:12.467746 25617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:41295 (error 108)
I20260805 23:08:12.467777 25616 catalog_manager.cc:4841] TS d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123): tablet bb75f9508e344116bc17d7678161de32 (table test-table [id=e02d0fd59d8045c486eb2156edf127b0]) successfully deleted
I20260805 23:08:12.467962 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 25925
I20260805 23:08:12.469033 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:12.468647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:41295 (local address 127.0.0.1:48180)
0805 23:08:12.468873 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:12.468936 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260805 23:08:12.470815 25617 catalog_manager.cc:4584] TS 742a0c3bcf0249fda5da0ab1d4bc2b37 (127.3.12.65:41295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb75f9508e344116bc17d7678161de32: Network error: Client connection negotiation failed: client connection to 127.3.12.65:41295: connect: Connection refused (error 111)
I20260805 23:08:12.485282 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 26128
I20260805 23:08:12.502254 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 25598
I20260805 23:08:32.518496 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40745
--webserver_interface=127.3.12.126
--webserver_port=46861
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:08:32.637956 26421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:32.638289 26421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:32.638365 26421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:32.642567 26421 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260805 23:08:32.642668 26421 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260805 23:08:32.642696 26421 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260805 23:08:32.642735 26421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:08:32.642802 26421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:32.642863 26421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:32.642903 26421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:08:32.642936 26421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:08:32.648551 26421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=46861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:32.649730 26421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:32.650799 26421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:08:32.665856 26421 server_base.cc:1034] running on GCE node
W20260805 23:08:32.665776 26428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:32.665741 26427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:32.665766 26431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:32.666702 26421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:32.667908 26421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:32.669095 26421 hybrid_clock.cc:648] HybridClock initialized: now 1785971312669052 us; error 58 us; skew 500 ppm
Aug 05 23:08:32 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971312, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:32.672135 26421 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:08:32.673096 26421 webserver.cc:460] Webserver started at http://127.3.12.126:46861/ using document root <none> and password file <none>
I20260805 23:08:32.673732 26421 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:32.673846 26421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:32.677969 26421 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260805 23:08:32.680383 26445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:32.681766 26421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:08:32.681927 26421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "d1a6e904d8bd46da805ae38dace975ad"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "d3342164c422ca7c877470c5f44685f0"
server_key_iv: "c62fafbd2f2f407bea76308ec6381b58"
server_key_version: "encryptionkey@0"
I20260805 23:08:32.682583 26421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:32.706535 26421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:32.709789 26421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:32.710006 26421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:32.717530 26421 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:40745
I20260805 23:08:32.717847 26523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:40745 every 8 connection(s)
I20260805 23:08:32.718742 26421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:08:32.723281 26524 sys_catalog.cc:263] Verifying existing consensus state
I20260805 23:08:32.725138 26524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Bootstrap starting.
I20260805 23:08:32.728473 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 26421
I20260805 23:08:32.728922 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:41295
--local_ip_for_outbound_sockets=127.3.12.65
--tserver_master_addrs=127.3.12.126:40745
--webserver_port=37455
--webserver_interface=127.3.12.65
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:08:32.742995 26524 log.cc:824] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:32.752121 26524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:08:32.752591 26524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Bootstrap complete.
I20260805 23:08:32.756817 26524 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:32.757253 26524 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a6e904d8bd46da805ae38dace975ad, State: Initialized, Role: FOLLOWER
I20260805 23:08:32.757761 26524 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:32.757915 26524 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:08:32.758015 26524 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:08:32.758141 26524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 1 FOLLOWER]: Advancing to term 2
I20260805 23:08:32.760087 26524 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:32.760468 26524 leader_election.cc:302] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a6e904d8bd46da805ae38dace975ad; no voters:
I20260805 23:08:32.760838 26524 leader_election.cc:288] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260805 23:08:32.760936 26536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 2 FOLLOWER]: Leader election won for term 2
I20260805 23:08:32.761164 26536 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 2 LEADER]: Becoming Leader. State: Replica: d1a6e904d8bd46da805ae38dace975ad, State: Running, Role: LEADER
I20260805 23:08:32.761462 26536 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:32.762128 26524 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:08:32.762447 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a6e904d8bd46da805ae38dace975ad. Latest consensus state: current_term: 2 leader_uuid: "d1a6e904d8bd46da805ae38dace975ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } } }
I20260805 23:08:32.762575 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: This master's current role is: LEADER
I20260805 23:08:32.762939 26542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:08:32.762974 26537 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d1a6e904d8bd46da805ae38dace975ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } } }
I20260805 23:08:32.763199 26537 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: This master's current role is: LEADER
I20260805 23:08:32.767920 26542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:08:32.768636 26542 catalog_manager.cc:1259] Loaded cluster ID: 087fe85f9a1845fc964846c0bfcaa416
I20260805 23:08:32.768731 26542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:08:32.770486 26542 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:08:32.771126 26542 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Loaded TSK: 0
I20260805 23:08:32.784507 26542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Generated new TSK 1
I20260805 23:08:32.784808 26542 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 23:08:32.855887 26529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:32.856205 26529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:32.856328 26529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:32.860450 26529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:32.860531 26529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:32.860625 26529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:08:32.866340 26529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:41295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=37455
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:32.867393 26529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:32.868491 26529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:32.878497 26567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:32.878556 26566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:32.878567 26569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:32.878928 26529 server_base.cc:1034] running on GCE node
I20260805 23:08:32.879364 26529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:32.880064 26529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:32.881259 26529 hybrid_clock.cc:648] HybridClock initialized: now 1785971312881231 us; error 47 us; skew 500 ppm
Aug 05 23:08:32 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971312, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:32.889142 26529 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:08:32.890368 26529 webserver.cc:460] Webserver started at http://127.3.12.65:37455/ using document root <none> and password file <none>
I20260805 23:08:32.891144 26529 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:32.891268 26529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:32.896178 26529 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260805 23:08:32.899508 26579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:32.900993 26529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260805 23:08:32.901145 26529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "00d3a98a48cea03dcab112078bd5107b"
server_key_iv: "54a6e6c91c5cdbcb1c0bed33867d0f7d"
server_key_version: "encryptionkey@0"
I20260805 23:08:32.901793 26529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:32.913187 26529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:32.916412 26529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:32.916628 26529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:32.917325 26529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:32.918496 26587 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260805 23:08:32.922837 26529 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260805 23:08:32.922906 26529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260805 23:08:32.922983 26529 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260805 23:08:32.924823 26529 ts_tablet_manager.cc:610] Registered 1 tablets
I20260805 23:08:32.924885 26529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260805 23:08:32.925159 26587 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Bootstrap starting.
I20260805 23:08:32.937335 26529 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:41295
I20260805 23:08:32.937359 26747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:41295 every 8 connection(s)
I20260805 23:08:32.938684 26529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:08:32.947063 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 26529
I20260805 23:08:32.947546 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:33093
--local_ip_for_outbound_sockets=127.3.12.66
--tserver_master_addrs=127.3.12.126:40745
--webserver_port=38103
--webserver_interface=127.3.12.66
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:08:32.948721 26587 log.cc:824] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:08:32 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971312, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:08:32.957254 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:32.941151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:37325 (local address 127.3.12.126:40745)
0805 23:08:32.941525 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:32.941558 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:32.941882 (+ 324us) server_negotiation.cc:407] Connection header received
0805 23:08:32.943002 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:32.943012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:32.943300 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:32.943639 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:32.944819 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:32.945800 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:32.947101 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:32.947852 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:32.948070 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:32.951553 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:32.951559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:32.951573 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:32.951605 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:32.953742 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:32.954331 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:32.954336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:32.954341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:32.954466 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:32.954850 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:32.954854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:32.954856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:32.955234 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:32.955396 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:32.956102 (+ 706us) server_negotiation.cc:299] Negotiation successful
0805 23:08:32.956410 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":125,"threads_started":1}
I20260805 23:08:32.957273 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:32.941177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:37325)
0805 23:08:32.941711 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:32.941754 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:32.942579 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:32.943692 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:32.943699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:32.944131 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:32.944640 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:32.944654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:32.945973 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:32.945975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:32.946935 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:32.946949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:32.948017 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:32.948024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:32.948350 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:32.949129 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:32.949154 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:32.951325 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:32.953879 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:32.953885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:32.953900 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:32.954189 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:32.954577 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:32.954581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:32.954583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:32.954740 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:32.955367 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:32.955374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:32.955660 (+ 286us) client_negotiation.cc:769] Sending connection context
0805 23:08:32.955940 (+ 280us) client_negotiation.cc:240] Negotiation successful
0805 23:08:32.956330 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":146,"threads_started":1}
I20260805 23:08:32.958597 26587 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:08:32.959105 26587 tablet_bootstrap.cc:492] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Bootstrap complete.
I20260805 23:08:32.960031 26748 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40745
I20260805 23:08:32.960319 26748 heartbeater.cc:464] Registering TS with master...
I20260805 23:08:32.960846 26587 ts_tablet_manager.cc:1397] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Time spent bootstrapping tablet: real 0.036s user 0.023s sys 0.012s
I20260805 23:08:32.960951 26748 heartbeater.cc:511] Master 127.3.12.126:40745 requested a full tablet report, sending...
I20260805 23:08:32.963599 26587 raft_consensus.cc:348] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:32.964196 26587 raft_consensus.cc:729] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742a0c3bcf0249fda5da0ab1d4bc2b37, State: Initialized, Role: FOLLOWER
I20260805 23:08:32.964756 26587 consensus_queue.cc:260] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:32.965341 26468 ts_manager.cc:194] Registered new tserver with Master: 742a0c3bcf0249fda5da0ab1d4bc2b37 (127.3.12.65:41295)
I20260805 23:08:32.965915 26587 ts_tablet_manager.cc:1428] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260805 23:08:32.969141 26468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:37325
I20260805 23:08:32.971952 26748 heartbeater.cc:503] Master 127.3.12.126:40745 was elected leader, sending a full tablet report...
Aug 05 23:08:32 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971312, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.65@KRBTEST.COM
I20260805 23:08:32.979887 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:32.968475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:41295 (local address 127.0.0.1:50460)
0805 23:08:32.969279 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:32.969327 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:32.969489 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:32.970132 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:32.970139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:32.970555 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:32.970669 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:32.970675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:32.971747 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:32.971752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:32.972765 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:32.972777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:32.973662 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:32.973666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:32.973787 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:32.974456 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:32.974471 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:32.975994 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:32.978148 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:32.978158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:32.978162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:32.978445 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:32.978780 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:32.978783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:32.978785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:32.978874 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:32.979384 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:32.979389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:32.979489 (+ 100us) client_negotiation.cc:769] Sending connection context
0805 23:08:32.979608 (+ 119us) client_negotiation.cc:240] Negotiation successful
0805 23:08:32.979760 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":119,"threads_started":1}
I20260805 23:08:32.980285 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:32.968554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.3.12.65:41295)
0805 23:08:32.968979 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:32.968993 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:32.969439 (+ 446us) server_negotiation.cc:407] Connection header received
0805 23:08:32.969675 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:32.969681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:32.969879 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:32.970014 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:32.970811 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:32.971613 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:32.972901 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:32.973436 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:32.973547 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:32.976137 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:32.976146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:32.976159 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:32.976185 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:32.978028 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:32.978563 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:32.978569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:32.978570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:32.978666 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:32.978968 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:32.978971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:32.978973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:32.979271 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:32.979399 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:32.979711 (+ 312us) server_negotiation.cc:299] Negotiation successful
0805 23:08:32.979878 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":141,"threads_started":1}
I20260805 23:08:32.981201 26649 tablet_service.cc:1514] Processing DeleteTablet for tablet bb75f9508e344116bc17d7678161de32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:08:12 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:50460
I20260805 23:08:32.982005 26763 tablet_replica.cc:330] stopping tablet replica
I20260805 23:08:32.982301 26763 raft_consensus.cc:2229] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:08:32.982579 26763 raft_consensus.cc:2258] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:08:32.983546 26763 ts_tablet_manager.cc:1905] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:08:32.987334 26763 ts_tablet_manager.cc:1918] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:08:32.987454 26763 log.cc:1196] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal/wals/bb75f9508e344116bc17d7678161de32
I20260805 23:08:32.987761 26763 ts_tablet_manager.cc:1939] T bb75f9508e344116bc17d7678161de32 P 742a0c3bcf0249fda5da0ab1d4bc2b37: Deleting consensus metadata
I20260805 23:08:32.989516 26448 catalog_manager.cc:4841] TS 742a0c3bcf0249fda5da0ab1d4bc2b37 (127.3.12.65:41295): tablet bb75f9508e344116bc17d7678161de32 (table test-table [id=e02d0fd59d8045c486eb2156edf127b0]) successfully deleted
W20260805 23:08:33.072124 26756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:33.072458 26756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:33.072515 26756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:33.076790 26756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:33.076850 26756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:33.076911 26756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:08:33.082212 26756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:33093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=38103
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:33.083292 26756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:33.084410 26756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:33.094676 26772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:33.094668 26770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:33.094672 26768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:33.095081 26756 server_base.cc:1034] running on GCE node
I20260805 23:08:33.095491 26756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:33.096216 26756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:33.097435 26756 hybrid_clock.cc:648] HybridClock initialized: now 1785971313097415 us; error 33 us; skew 500 ppm
Aug 05 23:08:33 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:33.100427 26756 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:08:33.101264 26756 webserver.cc:460] Webserver started at http://127.3.12.66:38103/ using document root <none> and password file <none>
I20260805 23:08:33.101794 26756 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:33.101871 26756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:33.105801 26756 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:08:33.107898 26779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:33.109256 26756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:08:33.109380 26756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "2542b95e31e7482690c3246f515721fc"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "e0075bee5a2813662e67096a13c1b52b"
server_key_iv: "22e02e26c62ea3179f3dda05ac312b6e"
server_key_version: "encryptionkey@0"
I20260805 23:08:33.109915 26756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:33.125716 26756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:33.128811 26756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:33.129020 26756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:33.129650 26756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:33.130505 26756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:08:33.130555 26756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:33.130618 26756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:08:33.130653 26756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:33.139930 26756 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:33093
I20260805 23:08:33.139948 26942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:33093 every 8 connection(s)
I20260805 23:08:33.141024 26756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:08:33.146747 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 26756
I20260805 23:08:33.147212 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:44123
--local_ip_for_outbound_sockets=127.3.12.67
--tserver_master_addrs=127.3.12.126:40745
--webserver_port=46827
--webserver_interface=127.3.12.67
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 05 23:08:33 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:08:33.156040 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.142944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:38833 (local address 127.3.12.126:40745)
0805 23:08:33.143179 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.143184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.144024 (+ 840us) server_negotiation.cc:407] Connection header received
0805 23:08:33.145372 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.145375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.145427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.145508 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:33.146887 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.147639 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.149093 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.149363 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.149488 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.152195 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:33.152198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:33.152201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:33.152229 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:33.153862 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:33.154415 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:33.154419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:33.154421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:33.154511 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:33.154883 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:33.154888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:33.154890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:33.155093 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:33.155206 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.155722 (+ 516us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.155863 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260805 23:08:33.156909 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.143385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:38833)
0805 23:08:33.143854 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:33.143899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:33.145167 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:33.145725 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:33.145733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:33.146153 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:33.146683 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.146704 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.147800 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.147804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.148953 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.148965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.149535 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.149539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.149710 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.150452 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:33.150470 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:33.151988 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:33.154006 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:33.154012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:33.154028 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:33.154304 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:33.154615 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:33.154618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:33.154620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:33.154780 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:33.155206 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:33.155214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:33.155471 (+ 257us) client_negotiation.cc:769] Sending connection context
0805 23:08:33.155678 (+ 207us) client_negotiation.cc:240] Negotiation successful
0805 23:08:33.156016 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":117,"threads_started":1}
I20260805 23:08:33.158170 26944 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40745
I20260805 23:08:33.158475 26944 heartbeater.cc:464] Registering TS with master...
I20260805 23:08:33.159175 26944 heartbeater.cc:511] Master 127.3.12.126:40745 requested a full tablet report, sending...
I20260805 23:08:33.160391 26468 ts_manager.cc:194] Registered new tserver with Master: 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093)
I20260805 23:08:33.160979 26468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:38833
W20260805 23:08:33.268867 26947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:33.269122 26947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:33.269203 26947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:33.273247 26947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:33.273309 26947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:33.273422 26947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:08:33.278932 26947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:44123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=46827
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:33.280014 26947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:33.281111 26947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:33.291636 26954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:33.291694 26947 server_base.cc:1034] running on GCE node
W20260805 23:08:33.291639 26953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:33.291632 26956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:33.292330 26947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:33.293037 26947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:33.294209 26947 hybrid_clock.cc:648] HybridClock initialized: now 1785971313294194 us; error 27 us; skew 500 ppm
Aug 05 23:08:33 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:33.297109 26947 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:08:33.297925 26947 webserver.cc:460] Webserver started at http://127.3.12.67:46827/ using document root <none> and password file <none>
I20260805 23:08:33.298453 26947 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:33.298527 26947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:33.302414 26947 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260805 23:08:33.304495 26965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:33.305837 26947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:08:33.305965 26947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "d4184e56e8404a4e9a34becca764ec9e"
format_stamp: "Formatted at 2026-08-05 23:08:12 on dist-test-slave-2d19"
server_key: "72fa3efb922d0789ab53109344e82f12"
server_key_iv: "814df1abde6b32c2b92a6867ee128ac3"
server_key_version: "encryptionkey@0"
I20260805 23:08:33.306489 26947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:33.318740 26947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:33.321683 26947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:33.321884 26947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:33.322471 26947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:33.323285 26947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:08:33.323334 26947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:33.323374 26947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:08:33.323415 26947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:33.332441 26947 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:44123
I20260805 23:08:33.332522 27126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:44123 every 8 connection(s)
I20260805 23:08:33.333463 26947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:08:33.335531 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 26947
Aug 05 23:08:33 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:08:33.345178 27131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.336280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.3.12.126:40745)
0805 23:08:33.336518 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.336522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.336881 (+ 359us) server_negotiation.cc:407] Connection header received
0805 23:08:33.336933 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.336936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.336987 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.337088 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:33.337865 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.338665 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.339637 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.339870 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.340018 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.341495 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:33.341501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:33.341503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:33.341540 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:33.343233 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:33.343663 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:33.343667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:33.343669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:33.343759 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:33.344165 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:33.344167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:33.344169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:33.344395 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:33.344585 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.344828 (+ 243us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.344960 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":65,"threads_started":1}
I20260805 23:08:33.348552 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.335398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:51173 (local address 127.3.12.126:40745)
0805 23:08:33.335550 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.335554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.336755 (+ 1201us) server_negotiation.cc:407] Connection header received
0805 23:08:33.338258 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.338260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.338300 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.338370 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:33.339614 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.340244 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.341543 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.341752 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.341891 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.344618 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:33.344622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:33.344624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:33.344647 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:33.346417 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:33.346932 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:33.346936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:33.346938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:33.347036 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:33.347436 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:33.347440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:33.347442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:33.347623 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:33.347734 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.348239 (+ 505us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.348387 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:08:33.349397 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.335914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:51173)
0805 23:08:33.336578 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:33.336638 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:33.338008 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:33.338537 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:33.338546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:33.338984 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:33.339464 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.339477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.340401 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.340405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.341378 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.341387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.341928 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.341933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.342110 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.342681 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:33.342697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:33.344431 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:33.346540 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:33.346548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:33.346567 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:33.346824 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:33.347133 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:33.347139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:33.347141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:33.347345 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:33.347735 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:33.347745 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:33.348002 (+ 257us) client_negotiation.cc:769] Sending connection context
0805 23:08:33.348198 (+ 196us) client_negotiation.cc:240] Negotiation successful
0805 23:08:33.348567 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":150,"threads_started":1}
I20260805 23:08:33.350562 27127 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40745
I20260805 23:08:33.350777 27127 heartbeater.cc:464] Registering TS with master...
I20260805 23:08:33.351294 27127 heartbeater.cc:511] Master 127.3.12.126:40745 requested a full tablet report, sending...
I20260805 23:08:33.352438 26468 ts_manager.cc:194] Registered new tserver with Master: d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123)
I20260805 23:08:33.353058 26468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:51173
I20260805 23:08:33.359160 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:08:33.363456 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.359811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.3.12.126:40745)
0805 23:08:33.360038 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.360042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.360184 (+ 142us) server_negotiation.cc:407] Connection header received
0805 23:08:33.360340 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.360342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.360445 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.360539 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:08:33.360826 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.361391 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.362271 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.362499 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.362637 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.362933 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:08:33.363136 (+ 203us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0805 23:08:33.363356 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59260: token expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260805 23:08:33.363549 26750 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59260: token expired
W20260805 23:08:33.363976 26347 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.12.126:40745)
I20260805 23:08:33.371122 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.364441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.3.12.126:40745)
0805 23:08:33.364573 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.364576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.364589 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:08:33.364670 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.364672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.364706 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.364767 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:33.365352 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.365832 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.366580 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.366778 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.366856 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.368240 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:33.368243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:33.368245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:33.368318 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:33.369589 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:33.370013 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:33.370016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:33.370017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:33.370094 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:33.370329 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:33.370331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:33.370332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:33.370492 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:33.370592 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.370788 (+ 196us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.370878 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:08:33.373720 26468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:08:33.377025 26468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:08:33.384686 26649 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c (DEFAULT_TABLE table=test-table [id=4453156f286c4113af2f42dda36905d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:08:33.385389 26649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:08:33.388356 26587 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Bootstrap starting.
I20260805 23:08:33.389123 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.383570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:44123 (local address 127.0.0.1:45512)
0805 23:08:33.383674 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:33.383690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:33.383787 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:33.384842 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:33.384845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:33.384863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:33.384976 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.384983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.386317 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.386320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.387551 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.387569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.388673 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.388677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.388779 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.388833 (+ 54us) client_negotiation.cc:769] Sending connection context
0805 23:08:33.388890 (+ 57us) client_negotiation.cc:240] Negotiation successful
0805 23:08:33.388995 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260805 23:08:33.389267 26587 tablet_bootstrap.cc:654] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Neither blocks nor log segments found. Creating new log.
I20260805 23:08:33.389513 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.383828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:33093 (local address 127.0.0.1:48048)
0805 23:08:33.384381 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:33.384396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:33.384521 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:33.385325 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:33.385328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:33.385347 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:33.385452 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.385460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.386749 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.386752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.387921 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.387934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.389048 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.389052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.389159 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.389202 (+ 43us) client_negotiation.cc:769] Sending connection context
0805 23:08:33.389288 (+ 86us) client_negotiation.cc:240] Negotiation successful
0805 23:08:33.389366 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":136,"threads_started":1}
I20260805 23:08:33.389992 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.384187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.3.12.66:33093)
0805 23:08:33.384684 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.384697 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.384715 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:08:33.384803 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.384810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.385189 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.385351 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:33.385627 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.386618 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.388063 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.388911 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.389014 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.389160 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.389726 (+ 566us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.389843 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":151,"threads_started":1}
I20260805 23:08:33.390336 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.383895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.3.12.67:44123)
0805 23:08:33.384298 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.384313 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.384339 (+ 26us) server_negotiation.cc:407] Connection header received
0805 23:08:33.384421 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.384427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.384673 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.384853 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:33.385080 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.386183 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.387681 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.388531 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.389494 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.389856 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.389909 (+ 53us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.390018 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":146,"threads_started":1}
I20260805 23:08:33.390978 26587 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: No bootstrap required, opened a new log
I20260805 23:08:33.391073 26587 ts_tablet_manager.cc:1397] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260805 23:08:33.391510 26587 raft_consensus.cc:348] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.391650 26587 raft_consensus.cc:374] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:08:33.391690 26587 raft_consensus.cc:729] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742a0c3bcf0249fda5da0ab1d4bc2b37, State: Initialized, Role: FOLLOWER
I20260805 23:08:33.391816 26587 consensus_queue.cc:260] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.392086 26587 ts_tablet_manager.cc:1428] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260805 23:08:33.393281 27042 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c (DEFAULT_TABLE table=test-table [id=4453156f286c4113af2f42dda36905d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:08:33.393324 26848 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c (DEFAULT_TABLE table=test-table [id=4453156f286c4113af2f42dda36905d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:08:33.394343 27042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:08:33.394410 26848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:08:33.401098 27147 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: Bootstrap starting.
I20260805 23:08:33.401603 27148 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: Bootstrap starting.
I20260805 23:08:33.403311 27147 tablet_bootstrap.cc:654] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: Neither blocks nor log segments found. Creating new log.
I20260805 23:08:33.403954 27148 tablet_bootstrap.cc:654] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: Neither blocks nor log segments found. Creating new log.
I20260805 23:08:33.404570 27147 log.cc:824] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:33.404779 27148 log.cc:824] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:33.406579 27148 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: No bootstrap required, opened a new log
I20260805 23:08:33.406599 27147 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: No bootstrap required, opened a new log
I20260805 23:08:33.406780 27148 ts_tablet_manager.cc:1397] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 23:08:33.406812 27147 ts_tablet_manager.cc:1397] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:08:33.410176 27148 raft_consensus.cc:348] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.410410 27148 raft_consensus.cc:374] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:08:33.410238 27147 raft_consensus.cc:348] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.410475 27148 raft_consensus.cc:729] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2542b95e31e7482690c3246f515721fc, State: Initialized, Role: FOLLOWER
I20260805 23:08:33.410495 27147 raft_consensus.cc:374] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:08:33.410574 27147 raft_consensus.cc:729] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4184e56e8404a4e9a34becca764ec9e, State: Initialized, Role: FOLLOWER
I20260805 23:08:33.410969 27148 consensus_queue.cc:260] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.410989 27147 consensus_queue.cc:260] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.411644 26944 heartbeater.cc:503] Master 127.3.12.126:40745 was elected leader, sending a full tablet report...
I20260805 23:08:33.411653 27127 heartbeater.cc:503] Master 127.3.12.126:40745 was elected leader, sending a full tablet report...
I20260805 23:08:33.412032 27148 ts_tablet_manager.cc:1428] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260805 23:08:33.412032 27147 ts_tablet_manager.cc:1428] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260805 23:08:33.439960 26749 tablet.cc:2369] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:08:33.508522 27152 raft_consensus.cc:482] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:08:33.508751 27152 raft_consensus.cc:504] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.509954 27152 leader_election.cc:288] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123), 742a0c3bcf0249fda5da0ab1d4bc2b37 (127.3.12.65:41295)
I20260805 23:08:33.514185 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.510292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:37815 (local address 127.3.12.67:44123)
0805 23:08:33.510436 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.510443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.510460 (+ 17us) server_negotiation.cc:407] Connection header received
0805 23:08:33.510519 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.510524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.510575 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.510654 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:33.510987 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.511649 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.512931 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.513619 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.513734 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.513799 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.513984 (+ 185us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.514061 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 23:08:33.514185 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.510181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:44123 (local address 127.3.12.66:37815)
0805 23:08:33.510299 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:33.510317 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:33.510412 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:33.510697 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:33.510702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:33.510727 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:33.510856 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.510862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.511765 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.511768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.512806 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.512816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.513738 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.513742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.513852 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.513876 (+ 24us) client_negotiation.cc:769] Sending connection context
0805 23:08:33.513966 (+ 90us) client_negotiation.cc:240] Negotiation successful
0805 23:08:33.514044 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260805 23:08:33.514428 27154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.510290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:41295 (local address 127.3.12.66:48343)
0805 23:08:33.510651 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:33.510664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:33.510751 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:33.511181 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:33.511184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:33.511204 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:08:33.511321 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.511327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.512214 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.512217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.513140 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:33.513151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.513896 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.513900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:33.514010 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.514033 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:08:33.514103 (+ 70us) client_negotiation.cc:240] Negotiation successful
0805 23:08:33.514174 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"spinlock_wait_cycles":896,"thread_start_us":116,"threads_started":1}
I20260805 23:08:33.514752 27155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.510563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:48343 (local address 127.3.12.65:41295)
0805 23:08:33.510857 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.510862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.510876 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:08:33.510932 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.510936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.511015 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.511159 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:08:33.511435 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.512094 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.513267 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.513771 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.513893 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.514072 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.514115 (+ 43us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.514182 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":112,"threads_started":1}
I20260805 23:08:33.514916 27065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4a6fd579bf41d9b35c07db7e1e2c6c" candidate_uuid: "2542b95e31e7482690c3246f515721fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4184e56e8404a4e9a34becca764ec9e" is_pre_election: true
I20260805 23:08:33.515254 27065 raft_consensus.cc:2454] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2542b95e31e7482690c3246f515721fc in term 0.
I20260805 23:08:33.515267 26675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4a6fd579bf41d9b35c07db7e1e2c6c" candidate_uuid: "2542b95e31e7482690c3246f515721fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" is_pre_election: true
I20260805 23:08:33.515512 26675 raft_consensus.cc:2454] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2542b95e31e7482690c3246f515721fc in term 0.
I20260805 23:08:33.515777 26780 leader_election.cc:302] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2542b95e31e7482690c3246f515721fc, d4184e56e8404a4e9a34becca764ec9e; no voters:
I20260805 23:08:33.516083 27152 raft_consensus.cc:2781] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:08:33.516188 27152 raft_consensus.cc:482] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:08:33.516271 27152 raft_consensus.cc:3037] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:08:33.517366 27152 raft_consensus.cc:504] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.517750 27152 leader_election.cc:288] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [CANDIDATE]: Term 1 election: Requested vote from peers d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123), 742a0c3bcf0249fda5da0ab1d4bc2b37 (127.3.12.65:41295)
I20260805 23:08:33.518203 27065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4a6fd579bf41d9b35c07db7e1e2c6c" candidate_uuid: "2542b95e31e7482690c3246f515721fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4184e56e8404a4e9a34becca764ec9e"
I20260805 23:08:33.518199 26675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4a6fd579bf41d9b35c07db7e1e2c6c" candidate_uuid: "2542b95e31e7482690c3246f515721fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37"
I20260805 23:08:33.518342 27065 raft_consensus.cc:3037] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:08:33.518342 26675 raft_consensus.cc:3037] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:08:33.519660 27065 raft_consensus.cc:2454] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2542b95e31e7482690c3246f515721fc in term 1.
I20260805 23:08:33.519661 26675 raft_consensus.cc:2454] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2542b95e31e7482690c3246f515721fc in term 1.
I20260805 23:08:33.519979 26781 leader_election.cc:302] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2542b95e31e7482690c3246f515721fc, 742a0c3bcf0249fda5da0ab1d4bc2b37; no voters:
I20260805 23:08:33.520176 27152 raft_consensus.cc:2781] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:08:33.520454 27152 raft_consensus.cc:686] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 1 LEADER]: Becoming Leader. State: Replica: 2542b95e31e7482690c3246f515721fc, State: Running, Role: LEADER
I20260805 23:08:33.520851 27152 consensus_queue.cc:237] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:33.524510 26468 catalog_manager.cc:5495] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2542b95e31e7482690c3246f515721fc (127.3.12.66). New cstate: current_term: 1 leader_uuid: "2542b95e31e7482690c3246f515721fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } health_report { overall_health: UNKNOWN } } }
I20260805 23:08:33.546054 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.542399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.3.12.66:33093)
0805 23:08:33.542547 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:33.542552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:33.542567 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:08:33.542693 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:33.542695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:33.542753 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:33.542870 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:08:33.543117 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.543618 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.544537 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:33.544762 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:33.544875 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:33.545195 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:08:33.545613 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:08:33.545749 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:33.545816 (+ 67us) server_negotiation.cc:299] Negotiation successful
0805 23:08:33.545878 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 23:08:33.552197 27065 raft_consensus.cc:1261] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 1 FOLLOWER]: Refusing update from remote peer 2542b95e31e7482690c3246f515721fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:08:33.552194 26675 raft_consensus.cc:1261] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Refusing update from remote peer 2542b95e31e7482690c3246f515721fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:08:33.553057 27156 consensus_queue.cc:1035] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:08:33.553249 27152 consensus_queue.cc:1035] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:08:33.561589 27165 mvcc.cc:205] Tried to move back new op lower bound from 7315338500301770752 to 7315338500184608768. Current Snapshot: MvccSnapshot[applied={T|T < 7315338500301770752}]
I20260805 23:08:33.569060 26467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:08:33.569250 26467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:08:33.571767 26467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Sending DeleteTablet for 3 replicas of tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c
I20260805 23:08:33.572594 27042 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:08:33 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:45512
I20260805 23:08:33.572662 26649 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:08:33 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:50460
I20260805 23:08:33.573078 27179 tablet_replica.cc:330] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: stopping tablet replica
I20260805 23:08:33.573127 27180 tablet_replica.cc:330] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: stopping tablet replica
I20260805 23:08:33.573195 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 26529
I20260805 23:08:33.573390 27179 raft_consensus.cc:2229] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:08:33.573432 27180 raft_consensus.cc:2229] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:08:33.573714 27180 raft_consensus.cc:2258] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:08:33.579222 27180 ts_tablet_manager.cc:1905] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:08:33.581789 27180 ts_tablet_manager.cc:1918] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:08:33.581933 27180 log.cc:1196] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/2c4a6fd579bf41d9b35c07db7e1e2c6c
I20260805 23:08:33.582420 27180 ts_tablet_manager.cc:1939] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P d4184e56e8404a4e9a34becca764ec9e: Deleting consensus metadata
I20260805 23:08:33.584936 26446 catalog_manager.cc:4841] TS d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123): tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c (table test-table [id=4453156f286c4113af2f42dda36905d1]) successfully deleted
I20260805 23:08:33.589643 26848 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:08:33 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:48048
I20260805 23:08:33.590440 27182 tablet_replica.cc:330] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: stopping tablet replica
I20260805 23:08:33.590837 27182 raft_consensus.cc:2229] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:08:33.591236 27182 raft_consensus.cc:2258] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:08:33.592819 27182 ts_tablet_manager.cc:1905] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:08:33.595125 27182 ts_tablet_manager.cc:1918] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:08:33.595253 27182 log.cc:1196] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/2c4a6fd579bf41d9b35c07db7e1e2c6c
I20260805 23:08:33.595587 27182 ts_tablet_manager.cc:1939] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 2542b95e31e7482690c3246f515721fc: Deleting consensus metadata
I20260805 23:08:33.596889 26446 catalog_manager.cc:4841] TS 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093): tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c (table test-table [id=4453156f286c4113af2f42dda36905d1]) successfully deleted
W20260805 23:08:33.597326 26448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:41295 (error 108)
I20260805 23:08:33.597982 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 26756
I20260805 23:08:33.604610 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:33.598174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:41295 (local address 127.0.0.1:50470)
0805 23:08:33.604383 (+ 6209us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:33.604492 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:41295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6014}
W20260805 23:08:33.605862 26448 catalog_manager.cc:4584] TS 742a0c3bcf0249fda5da0ab1d4bc2b37 (127.3.12.65:41295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c4a6fd579bf41d9b35c07db7e1e2c6c: Network error: Client connection negotiation failed: client connection to 127.3.12.65:41295: connect: Connection refused (error 111)
I20260805 23:08:33.612673 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 26947
I20260805 23:08:33.623641 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 26421
I20260805 23:08:53.638913 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40745
--webserver_interface=127.3.12.126
--webserver_port=46861
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:08:53.754366 27186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:53.754635 27186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:53.754675 27186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:53.758759 27186 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260805 23:08:53.758816 27186 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260805 23:08:53.758832 27186 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260805 23:08:53.758847 27186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:08:53.758896 27186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:53.758931 27186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:53.758961 27186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:08:53.758991 27186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:08:53.764235 27186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=46861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:53.765373 27186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:53.766398 27186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:53.775281 27198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:53.775378 27194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:53.775285 27193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:53.775457 27186 server_base.cc:1034] running on GCE node
I20260805 23:08:53.776388 27186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:53.777515 27186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:53.778734 27186 hybrid_clock.cc:648] HybridClock initialized: now 1785971333778715 us; error 37 us; skew 500 ppm
Aug 05 23:08:53 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971333, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:53.787588 27186 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:08:53.788621 27186 webserver.cc:460] Webserver started at http://127.3.12.126:46861/ using document root <none> and password file <none>
I20260805 23:08:53.789172 27186 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:53.789235 27186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:53.793434 27186 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:08:53.795542 27210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:53.796859 27186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:08:53.797011 27186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "d1a6e904d8bd46da805ae38dace975ad"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "d3342164c422ca7c877470c5f44685f0"
server_key_iv: "c62fafbd2f2f407bea76308ec6381b58"
server_key_version: "encryptionkey@0"
I20260805 23:08:53.797559 27186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:53.814931 27186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:53.818192 27186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:53.818409 27186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:53.826097 27186 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:40745
I20260805 23:08:53.826431 27287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:40745 every 8 connection(s)
I20260805 23:08:53.827255 27186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:08:53.831616 27288 sys_catalog.cc:263] Verifying existing consensus state
I20260805 23:08:53.833470 27288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Bootstrap starting.
I20260805 23:08:53.837072 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 27186
I20260805 23:08:53.837600 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:41295
--local_ip_for_outbound_sockets=127.3.12.65
--tserver_master_addrs=127.3.12.126:40745
--webserver_port=37455
--webserver_interface=127.3.12.65
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:08:53.853523 27288 log.cc:824] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:53.865012 27288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:08:53.865422 27288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Bootstrap complete.
I20260805 23:08:53.869681 27288 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:53.870121 27288 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a6e904d8bd46da805ae38dace975ad, State: Initialized, Role: FOLLOWER
I20260805 23:08:53.870609 27288 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:53.870764 27288 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:08:53.870852 27288 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:08:53.870982 27288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 2 FOLLOWER]: Advancing to term 3
I20260805 23:08:53.872980 27288 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:53.873311 27288 leader_election.cc:302] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1a6e904d8bd46da805ae38dace975ad; no voters:
I20260805 23:08:53.873663 27288 leader_election.cc:288] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [CANDIDATE]: Term 3 election: Requested vote from peers
I20260805 23:08:53.873750 27294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 3 FOLLOWER]: Leader election won for term 3
I20260805 23:08:53.873972 27294 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [term 3 LEADER]: Becoming Leader. State: Replica: d1a6e904d8bd46da805ae38dace975ad, State: Running, Role: LEADER
I20260805 23:08:53.874341 27294 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } }
I20260805 23:08:53.874984 27288 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:08:53.875787 27295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d1a6e904d8bd46da805ae38dace975ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } } }
I20260805 23:08:53.875919 27295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: This master's current role is: LEADER
I20260805 23:08:53.876365 27297 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1a6e904d8bd46da805ae38dace975ad. Latest consensus state: current_term: 3 leader_uuid: "d1a6e904d8bd46da805ae38dace975ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1a6e904d8bd46da805ae38dace975ad" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40745 } } }
I20260805 23:08:53.876515 27297 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad [sys.catalog]: This master's current role is: LEADER
I20260805 23:08:53.876892 27310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:08:53.882198 27310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:08:53.882871 27310 catalog_manager.cc:1259] Loaded cluster ID: 087fe85f9a1845fc964846c0bfcaa416
I20260805 23:08:53.882936 27310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:08:53.886150 27310 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:08:53.886863 27310 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Loaded TSK: 0
I20260805 23:08:53.886948 27310 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d1a6e904d8bd46da805ae38dace975ad: Loaded TSK: 1
F20260805 23:08:53.887809 27310 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4048f089ad google::LogMessage::Fail() at ??:0
@ 0x7f4048f0b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4048f084de google::LogMessage::Flush() at ??:0
@ 0x7f4048f09a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f404bbcc18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f404bbc4f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f405386d7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f405384799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4053848178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4053889df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f40538a882b std::function<>::operator()() at ??:0
@ 0x7f4053847b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4053848f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4053845579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4053889640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f404b15a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f404b15ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f404b15c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f404b14b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f404c78f6db start_thread at ??:0
@ 0x7f4047a0661f clone at ??:0
W20260805 23:08:53.995812 27292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:53.996063 27292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:53.996115 27292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:54.000171 27292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:54.000228 27292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:54.000356 27292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:08:54.005651 27292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:41295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=37455
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:54.006769 27292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:54.007820 27292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:54.017911 27364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:54.017957 27365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:54.018006 27368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:54.018383 27292 server_base.cc:1034] running on GCE node
I20260805 23:08:54.018836 27292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:54.019517 27292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:54.020705 27292 hybrid_clock.cc:648] HybridClock initialized: now 1785971334020672 us; error 52 us; skew 500 ppm
Aug 05 23:08:54 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:54.023855 27292 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:08:54.024962 27292 webserver.cc:460] Webserver started at http://127.3.12.65:37455/ using document root <none> and password file <none>
I20260805 23:08:54.025583 27292 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:54.025650 27292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:54.029822 27292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:08:54.032194 27382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:54.033546 27292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:08:54.033736 27292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "00d3a98a48cea03dcab112078bd5107b"
server_key_iv: "54a6e6c91c5cdbcb1c0bed33867d0f7d"
server_key_version: "encryptionkey@0"
I20260805 23:08:54.034325 27292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:54.058348 27292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:54.061757 27292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:54.061966 27292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:54.062719 27292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:54.063865 27393 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260805 23:08:54.068167 27292 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260805 23:08:54.068238 27292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260805 23:08:54.068342 27292 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260805 23:08:54.070217 27292 ts_tablet_manager.cc:610] Registered 1 tablets
I20260805 23:08:54.070279 27292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260805 23:08:54.070537 27393 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Bootstrap starting.
I20260805 23:08:54.082132 27292 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:41295
I20260805 23:08:54.082363 27557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:41295 every 8 connection(s)
I20260805 23:08:54.083494 27292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:08:54.088315 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 27292
I20260805 23:08:54.088848 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:33093
--local_ip_for_outbound_sockets=127.3.12.66
--tserver_master_addrs=127.3.12.126:40745
--webserver_port=38103
--webserver_interface=127.3.12.66
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:08:54.093909 27393 log.cc:824] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Log is configured to *not* fsync() on all Append() calls
I20260805 23:08:54.101399 27393 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 23:08:54.101857 27393 tablet_bootstrap.cc:492] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Bootstrap complete.
I20260805 23:08:54.103091 27393 ts_tablet_manager.cc:1397] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.008s
I20260805 23:08:54.105851 27393 raft_consensus.cc:348] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:54.106315 27393 raft_consensus.cc:729] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742a0c3bcf0249fda5da0ab1d4bc2b37, State: Initialized, Role: FOLLOWER
I20260805 23:08:54.106825 27393 consensus_queue.cc:260] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:54.107807 27393 ts_tablet_manager.cc:1428] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260805 23:08:54.141705 27560 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:08:54.085914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:41695)
0805 23:08:54.086550 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.086616 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:54.087714 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:54.140887 (+ 53173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":171,"threads_started":1}
W20260805 23:08:54.142385 27558 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:40745 (0 consecutive failures): Network error: Failed to ping master at 127.3.12.126:40745: Client connection negotiation failed: client connection to 127.3.12.126:40745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260805 23:08:54.143193 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:54.142909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:32915)
0805 23:08:54.143042 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.143133 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260805 23:08:54.143937 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:54.143716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:45847)
0805 23:08:54.143825 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.143874 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260805 23:08:54.144160 27558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 23:08:54.221606 27561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:54.221922 27561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:54.222003 27561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:54.226130 27561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:54.226248 27561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:54.226351 27561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:08:54.232093 27561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:33093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=38103
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:54.233309 27561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:54.234426 27561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:54.244728 27570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:54.244868 27561 server_base.cc:1034] running on GCE node
W20260805 23:08:54.244750 27577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:54.244742 27574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:54.245476 27561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:54.246129 27561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:54.247323 27561 hybrid_clock.cc:648] HybridClock initialized: now 1785971334247294 us; error 50 us; skew 500 ppm
Aug 05 23:08:54 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:54.250133 27561 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:08:54.251086 27561 webserver.cc:460] Webserver started at http://127.3.12.66:38103/ using document root <none> and password file <none>
I20260805 23:08:54.251677 27561 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:54.251775 27561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:54.255649 27561 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:08:54.257840 27585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:54.259193 27561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:08:54.259330 27561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "2542b95e31e7482690c3246f515721fc"
format_stamp: "Formatted at 2026-08-05 23:08:11 on dist-test-slave-2d19"
server_key: "e0075bee5a2813662e67096a13c1b52b"
server_key_iv: "22e02e26c62ea3179f3dda05ac312b6e"
server_key_version: "encryptionkey@0"
I20260805 23:08:54.260028 27561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:54.279095 27561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:54.282223 27561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:54.282437 27561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:54.283067 27561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:54.283931 27561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:08:54.283979 27561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:54.284052 27561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:08:54.284088 27561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:54.293639 27561 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:33093
I20260805 23:08:54.293668 27765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:33093 every 8 connection(s)
I20260805 23:08:54.294695 27561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:08:54.298312 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 27561
I20260805 23:08:54.298653 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:54.296865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:39403)
0805 23:08:54.297328 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.297435 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":125,"threads_started":1}
I20260805 23:08:54.298792 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:44123
--local_ip_for_outbound_sockets=127.3.12.67
--tserver_master_addrs=127.3.12.126:40745
--webserver_port=46827
--webserver_interface=127.3.12.67
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:08:54.299149 27766 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:40745 (0 consecutive failures): Network error: Failed to ping master at 127.3.12.126:40745: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
I20260805 23:08:54.299809 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:54.299595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:50873)
0805 23:08:54.299709 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.299756 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260805 23:08:54.300488 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:54.300278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:32853)
0805 23:08:54.300401 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.300442 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260805 23:08:54.300747 27766 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260805 23:08:54.411685 27770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:08:54.411957 27770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:08:54.412001 27770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:08:54.416126 27770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:08:54.416181 27770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:08:54.416294 27770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:08:54.421487 27770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:44123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=46827
--tserver_master_addrs=127.3.12.126:40745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:08:54.422545 27770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:08:54.423576 27770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:08:54.433436 27781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:54.433446 27778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:08:54.433445 27779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:08:54.433723 27770 server_base.cc:1034] running on GCE node
I20260805 23:08:54.434263 27770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:08:54.434854 27770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:08:54.436049 27770 hybrid_clock.cc:648] HybridClock initialized: now 1785971334436035 us; error 31 us; skew 500 ppm
Aug 05 23:08:54 dist-test-slave-2d19 krb5kdc[25580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:08:54.439088 27770 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:08:54.440021 27770 webserver.cc:460] Webserver started at http://127.3.12.67:46827/ using document root <none> and password file <none>
I20260805 23:08:54.440670 27770 fs_manager.cc:359] Metadata directory not provided
I20260805 23:08:54.440733 27770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:08:54.445107 27770 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:08:54.447247 27792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:54.448585 27770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:08:54.448711 27770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "d4184e56e8404a4e9a34becca764ec9e"
format_stamp: "Formatted at 2026-08-05 23:08:12 on dist-test-slave-2d19"
server_key: "72fa3efb922d0789ab53109344e82f12"
server_key_iv: "814df1abde6b32c2b92a6867ee128ac3"
server_key_version: "encryptionkey@0"
I20260805 23:08:54.449272 27770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:08:54.469105 27770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:08:54.472067 27770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:08:54.472309 27770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:08:54.472920 27770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:08:54.473748 27770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:08:54.473794 27770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:54.473862 27770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:08:54.473901 27770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:08:54.483078 27770 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:44123
I20260805 23:08:54.483109 27962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:44123 every 8 connection(s)
I20260805 23:08:54.484069 27770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:08:54.487079 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 27770
I20260805 23:08:54.488176 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:54.486321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:35501)
0805 23:08:54.486792 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.486897 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
W20260805 23:08:54.488693 27963 heartbeater.cc:644] Failed to heartbeat to 127.3.12.126:40745 (0 consecutive failures): Network error: Failed to ping master at 127.3.12.126:40745: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
I20260805 23:08:54.489241 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:54.489042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:40153)
0805 23:08:54.489141 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.489187 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260805 23:08:54.489830 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:54.489651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:50029)
0805 23:08:54.489743 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:54.489782 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260805 23:08:54.490006 27963 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260805 23:08:55.145218 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:55.144800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:36719)
0805 23:08:55.145053 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:55.145108 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260805 23:08:55.301750 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:55.301315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:47083)
0805 23:08:55.301580 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:55.301651 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
I20260805 23:08:55.364959 27972 raft_consensus.cc:482] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:08:55.365228 27972 raft_consensus.cc:504] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:55.366420 27972 leader_election.cc:288] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123), 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093)
Aug 05 23:08:55 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.66@KRBTEST.COM
Aug 05 23:08:55 dist-test-slave-2d19 krb5kdc[25580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.67@KRBTEST.COM
I20260805 23:08:55.384032 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:55.366868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:60497 (local address 127.3.12.67:44123)
0805 23:08:55.367290 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:55.367306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:55.367355 (+ 49us) server_negotiation.cc:407] Connection header received
0805 23:08:55.368311 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:55.368321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:55.368796 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:55.369150 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:55.370990 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:55.372671 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:55.374661 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:55.375391 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:55.375576 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:55.379386 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:55.379391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:55.379402 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:55.379432 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:55.381343 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:55.381940 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:55.381945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:55.381949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:55.382059 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:55.382463 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:55.382466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:55.382467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:55.382865 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:55.383007 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:55.383573 (+ 566us) server_negotiation.cc:299] Negotiation successful
0805 23:08:55.383842 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260805 23:08:55.384018 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:55.366995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:33093 (local address 127.3.12.65:58951)
0805 23:08:55.367357 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:55.367374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:55.367475 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:55.369259 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:55.369265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:55.369930 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:55.370759 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:55.370781 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:55.372851 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:55.372855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:55.374527 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:55.374539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:55.375534 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:55.375537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:55.375847 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:55.376797 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:55.376824 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:55.378822 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:55.381379 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:55.381389 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:55.381407 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:55.381753 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:55.382166 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:55.382170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:55.382172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:55.382329 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:55.383000 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:55.383006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:55.383259 (+ 253us) client_negotiation.cc:769] Sending connection context
0805 23:08:55.383460 (+ 201us) client_negotiation.cc:240] Negotiation successful
0805 23:08:55.383832 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":370,"spinlock_wait_cycles":16256,"thread_start_us":69,"threads_started":1}
I20260805 23:08:55.384083 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:55.367155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:58951 (local address 127.3.12.66:33093)
0805 23:08:55.367452 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0805 23:08:55.367477 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0805 23:08:55.367515 (+ 38us) server_negotiation.cc:407] Connection header received
0805 23:08:55.368323 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:08:55.368330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:08:55.368796 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:08:55.369150 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:08:55.370990 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:55.372671 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:55.374661 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:08:55.375391 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:08:55.375576 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:55.379147 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:08:55.379155 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:08:55.379174 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:08:55.379210 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:08:55.381245 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:55.381871 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:55.381880 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:55.381889 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:55.382039 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:08:55.382429 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:08:55.382434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:08:55.382436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:08:55.382866 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:08:55.383014 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 23:08:55.383576 (+ 562us) server_negotiation.cc:299] Negotiation successful
0805 23:08:55.383876 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":63,"threads_started":1}
I20260805 23:08:55.384972 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:55.366694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:44123 (local address 127.3.12.65:60497)
0805 23:08:55.366859 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:55.366878 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0805 23:08:55.367010 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:08:55.369259 (+ 2249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:08:55.369265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:08:55.369930 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:08:55.370759 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:55.370781 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:55.372851 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:55.372855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:55.374526 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:08:55.374539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:08:55.375534 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:08:55.375537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:08:55.375848 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:08:55.376800 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:08:55.376838 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:08:55.379190 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:08:55.381477 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:55.381484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:55.381489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:55.381825 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:55.382197 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:08:55.382200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:08:55.382201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:08:55.382344 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:08:55.383000 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:08:55.383006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:08:55.383259 (+ 253us) client_negotiation.cc:769] Sending connection context
0805 23:08:55.383460 (+ 201us) client_negotiation.cc:240] Negotiation successful
0805 23:08:55.383932 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":577,"spinlock_wait_cycles":74880}
W20260805 23:08:55.386854 27385 leader_election.cc:341] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093): Not found: Tablet not found: 2c4a6fd579bf41d9b35c07db7e1e2c6c
W20260805 23:08:55.387169 27385 leader_election.cc:341] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123): Not found: Tablet not found: 2c4a6fd579bf41d9b35c07db7e1e2c6c
I20260805 23:08:55.387240 27385 leader_election.cc:302] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 742a0c3bcf0249fda5da0ab1d4bc2b37; no voters: 2542b95e31e7482690c3246f515721fc, d4184e56e8404a4e9a34becca764ec9e
I20260805 23:08:55.387523 27972 raft_consensus.cc:2726] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:08:55.491129 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:55.490697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:54005)
0805 23:08:55.490974 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:55.491031 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260805 23:08:56.146453 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:56.146034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:37177)
0805 23:08:56.146294 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:56.146351 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260805 23:08:56.302875 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:56.302473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:37967)
0805 23:08:56.302727 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:56.302785 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260805 23:08:56.492322 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:56.491854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:48589)
0805 23:08:56.492131 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:56.492189 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260805 23:08:57.147660 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:57.147253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:55793)
0805 23:08:57.147505 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:57.147559 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
I20260805 23:08:57.304131 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:57.303684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:60091)
0805 23:08:57.303960 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:57.304016 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260805 23:08:57.368230 27988 raft_consensus.cc:482] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:08:57.368443 27988 raft_consensus.cc:504] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:08:57.368896 27988 leader_election.cc:288] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123), 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093)
W20260805 23:08:57.369984 27385 leader_election.cc:341] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123): Not found: Tablet not found: 2c4a6fd579bf41d9b35c07db7e1e2c6c
W20260805 23:08:57.370168 27385 leader_election.cc:341] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093): Not found: Tablet not found: 2c4a6fd579bf41d9b35c07db7e1e2c6c
I20260805 23:08:57.370244 27385 leader_election.cc:302] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 742a0c3bcf0249fda5da0ab1d4bc2b37; no voters: 2542b95e31e7482690c3246f515721fc, d4184e56e8404a4e9a34becca764ec9e
I20260805 23:08:57.370448 27988 raft_consensus.cc:2726] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:08:57.493430 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:57.493020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:57069)
0805 23:08:57.493286 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:57.493337 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
I20260805 23:08:58.149047 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:58.148553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:54149)
0805 23:08:58.148834 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:58.148924 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
I20260805 23:08:58.305375 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:58.304968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:32869)
0805 23:08:58.305228 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:58.305282 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260805 23:08:58.494724 27997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:58.494255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:56943)
0805 23:08:58.494546 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:58.494615 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260805 23:08:59.150484 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:59.150015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:37727)
0805 23:08:59.150326 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:59.150387 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":82,"threads_started":1}
I20260805 23:08:59.306747 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:59.306259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:37401)
0805 23:08:59.306579 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:59.306644 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260805 23:08:59.496177 28000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:08:59.495672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:60849)
0805 23:08:59.495990 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0805 23:08:59.496064 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
I20260805 23:09:00.151803 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:00.151338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:32789)
0805 23:09:00.151605 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:00.151700 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
I20260805 23:09:00.308194 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:00.307686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:47453)
0805 23:09:00.308005 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:00.308068 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260805 23:09:00.423470 28003 raft_consensus.cc:482] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:09:00.423635 28003 raft_consensus.cc:504] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:09:00.424077 28003 leader_election.cc:288] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123), 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093)
W20260805 23:09:00.424971 27385 leader_election.cc:341] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123): Not found: Tablet not found: 2c4a6fd579bf41d9b35c07db7e1e2c6c
W20260805 23:09:00.425096 27385 leader_election.cc:341] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093): Not found: Tablet not found: 2c4a6fd579bf41d9b35c07db7e1e2c6c
I20260805 23:09:00.425134 27385 leader_election.cc:302] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 742a0c3bcf0249fda5da0ab1d4bc2b37; no voters: 2542b95e31e7482690c3246f515721fc, d4184e56e8404a4e9a34becca764ec9e
I20260805 23:09:00.425273 28003 raft_consensus.cc:2726] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:09:00.497440 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:00.497026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:47781)
0805 23:09:00.497292 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:00.497344 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260805 23:09:01.153190 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:01.152612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:42831)
0805 23:09:01.152864 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:01.153034 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260805 23:09:01.309484 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:01.308992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:38759)
0805 23:09:01.309322 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:01.309378 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":97,"threads_started":1}
I20260805 23:09:01.498759 28009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:01.498264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:58029)
0805 23:09:01.498585 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:01.498662 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":82,"threads_started":1}
I20260805 23:09:02.154561 28010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:02.154070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:59979)
0805 23:09:02.154368 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:02.154446 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260805 23:09:02.310794 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:02.310273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:38733)
0805 23:09:02.310603 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:02.310681 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260805 23:09:02.500111 28012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:02.499608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:60109)
0805 23:09:02.499880 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:02.499998 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260805 23:09:03.155781 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:03.155344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:60225)
0805 23:09:03.155629 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:03.155684 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
I20260805 23:09:03.312141 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:03.311717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:54163)
0805 23:09:03.311982 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:03.312045 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260805 23:09:03.501479 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:03.501010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:59985)
0805 23:09:03.501312 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:03.501376 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":87,"threads_started":1}
I20260805 23:09:04.157160 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:04.156728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:36019)
0805 23:09:04.156999 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:04.157061 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260805 23:09:04.313413 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:04.312973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:47427)
0805 23:09:04.313255 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:04.313315 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260805 23:09:04.502889 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:04.502379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:36353)
0805 23:09:04.502723 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:04.502788 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":128,"threads_started":1}
I20260805 23:09:05.158381 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:05.157956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:36421)
0805 23:09:05.158218 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:05.158283 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260805 23:09:05.314698 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:05.314238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:34589)
0805 23:09:05.314526 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:05.314583 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":115,"threads_started":1}
I20260805 23:09:05.455435 28024 raft_consensus.cc:482] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:09:05.455621 28024 raft_consensus.cc:504] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4184e56e8404a4e9a34becca764ec9e" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44123 } } peers { permanent_uuid: "2542b95e31e7482690c3246f515721fc" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 33093 } } peers { permanent_uuid: "742a0c3bcf0249fda5da0ab1d4bc2b37" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41295 } }
I20260805 23:09:05.456096 28024 leader_election.cc:288] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123), 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093)
W20260805 23:09:05.457018 27385 leader_election.cc:341] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d4184e56e8404a4e9a34becca764ec9e (127.3.12.67:44123): Not found: Tablet not found: 2c4a6fd579bf41d9b35c07db7e1e2c6c
W20260805 23:09:05.457183 27385 leader_election.cc:341] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2542b95e31e7482690c3246f515721fc (127.3.12.66:33093): Not found: Tablet not found: 2c4a6fd579bf41d9b35c07db7e1e2c6c
I20260805 23:09:05.457248 27385 leader_election.cc:302] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 742a0c3bcf0249fda5da0ab1d4bc2b37; no voters: 2542b95e31e7482690c3246f515721fc, d4184e56e8404a4e9a34becca764ec9e
I20260805 23:09:05.457441 28024 raft_consensus.cc:2726] T 2c4a6fd579bf41d9b35c07db7e1e2c6c P 742a0c3bcf0249fda5da0ab1d4bc2b37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 23:09:05.504319 28025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:05.503810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:52739)
0805 23:09:05.504100 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:05.504160 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260805 23:09:06.159614 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:06.159181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:55733)
0805 23:09:06.159453 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:06.159514 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":102,"threads_started":1}
I20260805 23:09:06.315920 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:06.315456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:37133)
0805 23:09:06.315765 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:06.315821 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":85,"threads_started":1}
I20260805 23:09:06.505667 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:06.505223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:48663)
0805 23:09:06.505497 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:06.505566 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
I20260805 23:09:07.160979 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:07.160487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:41015)
0805 23:09:07.160811 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:07.160874 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
I20260805 23:09:07.317096 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:07.316697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:57301)
0805 23:09:07.316945 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:07.316998 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260805 23:09:07.506938 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:07.506545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:43129)
0805 23:09:07.506785 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:07.506840 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260805 23:09:08.162575 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:08.161941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:55873)
0805 23:09:08.162273 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:08.162334 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260805 23:09:08.318322 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:08.317856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:35673)
0805 23:09:08.318164 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:08.318222 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":83,"threads_started":1}
I20260805 23:09:08.508294 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:08.507799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:42147)
0805 23:09:08.508101 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:08.508165 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
I20260805 23:09:09.164055 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:09.163559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.65:44959)
0805 23:09:09.163832 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:09.163890 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260805 23:09:09.319623 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:09.319213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.66:40381)
0805 23:09:09.319473 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:09.319525 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260805 23:09:09.491879 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 27292
I20260805 23:09:09.502208 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 27561
I20260805 23:09:09.509863 28038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:09.509423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40745 (local address 127.3.12.67:53981)
0805 23:09:09.509709 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:09.509766 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.126:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260805 23:09:09.513779 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 27770
2026-08-05T23:09:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59562 ms)
[----------] 30 tests from SecurityITest (254030 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:09 dist-test-slave-2d19 krb5kdc[28042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:09 dist-test-slave-2d19 krb5kdc[28042](info): set up 2 sockets
Aug 05 23:09:09 dist-test-slave-2d19 krb5kdc[28042](info): commencing operation
krb5kdc: starting...
W20260805 23:09:10.750073 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:10 dist-test-slave-2d19 krb5kdc[28042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:10.887708 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37667
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:37667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:11.001222 28067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:11.001550 28067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:11.001626 28067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:11.005594 28067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:11.005709 28067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:11.005749 28067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:11.005789 28067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:11.005828 28067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:11.011104 28067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:37667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:11.012607 28067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:11.013705 28067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:11.022971 28077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:11.023090 28067 server_base.cc:1034] running on GCE node
W20260805 23:09:11.023017 28074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:11.023021 28073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:11.023975 28067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:11.025189 28067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:11.026386 28067 hybrid_clock.cc:648] HybridClock initialized: now 1785971351026367 us; error 34 us; skew 500 ppm
Aug 05 23:09:11 dist-test-slave-2d19 krb5kdc[28042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:11.029599 28067 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:11.030639 28067 webserver.cc:460] Webserver started at http://127.3.12.126:32955/ using document root <none> and password file <none>
I20260805 23:09:11.031251 28067 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:11.031320 28067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:11.031569 28067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:11.033874 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "b8bca580adfb4a84b07cd2fbdda04fc7"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "a1b86387d889e16e1661605e35d19dea"
server_key_iv: "67be395bb95e96892758d0a70c0c47ef"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.034510 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "b8bca580adfb4a84b07cd2fbdda04fc7"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "a1b86387d889e16e1661605e35d19dea"
server_key_iv: "67be395bb95e96892758d0a70c0c47ef"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.038800 28067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:11.041460 28090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.043057 28067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260805 23:09:11.043238 28067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "b8bca580adfb4a84b07cd2fbdda04fc7"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "a1b86387d889e16e1661605e35d19dea"
server_key_iv: "67be395bb95e96892758d0a70c0c47ef"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.043442 28067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:11.073513 28067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:11.077165 28067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:11.077348 28067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:11.085196 28067 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:37667
I20260805 23:09:11.085244 28163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:37667 every 8 connection(s)
I20260805 23:09:11.086270 28067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:11.089237 28164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:11.094806 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7: Bootstrap starting.
I20260805 23:09:11.095888 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 28067
I20260805 23:09:11.096030 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:11.096414 3121 external_mini_cluster.cc:1428] Setting key 8b9249adf2a3cb443c4b4a741ffbb7c0
I20260805 23:09:11.097560 28164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:11.098410 28164 log.cc:824] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:11.100481 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7: No bootstrap required, opened a new log
I20260805 23:09:11.103943 28164 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bca580adfb4a84b07cd2fbdda04fc7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37667 } }
I20260805 23:09:11.104202 28164 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:11.104343 28164 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8bca580adfb4a84b07cd2fbdda04fc7, State: Initialized, Role: FOLLOWER
I20260805 23:09:11.104779 28164 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bca580adfb4a84b07cd2fbdda04fc7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37667 } }
I20260805 23:09:11.104933 28164 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:11.105027 28164 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:11.105140 28164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 0 FOLLOWER]: Advancing to term 1
Aug 05 23:09:11 dist-test-slave-2d19 krb5kdc[28042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:11.106371 28164 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bca580adfb4a84b07cd2fbdda04fc7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37667 } }
I20260805 23:09:11.106748 28164 leader_election.cc:302] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8bca580adfb4a84b07cd2fbdda04fc7; no voters:
I20260805 23:09:11.107137 28164 leader_election.cc:288] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:11.107240 28170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:11.107533 28170 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [term 1 LEADER]: Becoming Leader. State: Replica: b8bca580adfb4a84b07cd2fbdda04fc7, State: Running, Role: LEADER
I20260805 23:09:11.107900 28170 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bca580adfb4a84b07cd2fbdda04fc7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37667 } }
I20260805 23:09:11.108471 28164 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:11.109246 28173 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8bca580adfb4a84b07cd2fbdda04fc7. Latest consensus state: current_term: 1 leader_uuid: "b8bca580adfb4a84b07cd2fbdda04fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bca580adfb4a84b07cd2fbdda04fc7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37667 } } }
I20260805 23:09:11.110085 28173 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:11.109946 28171 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8bca580adfb4a84b07cd2fbdda04fc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8bca580adfb4a84b07cd2fbdda04fc7" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 37667 } } }
I20260805 23:09:11.110426 28171 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:11.112474 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.097603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.3.12.126:37667)
0805 23:09:11.098062 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:11.098077 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:11.098112 (+ 35us) server_negotiation.cc:407] Connection header received
0805 23:09:11.098884 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:11.098908 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:11.099437 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:11.099813 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:11.100677 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.101663 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.102560 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.103071 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.103279 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.106395 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:11.106400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:11.106413 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:11.106440 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:11.108553 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.109110 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.109115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.109122 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.109285 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.109663 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.109667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.109671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.110257 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:11.110476 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:11.111269 (+ 793us) server_negotiation.cc:299] Negotiation successful
0805 23:09:11.111595 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":155,"threads_started":1}
W20260805 23:09:11.113612 28192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:11.113692 28192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:11.113788 28189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:11.114626 28189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:11.120002 28189 catalog_manager.cc:1347] Generated new cluster ID: b405edc72ad64504b02f4bdc716b3dd5
I20260805 23:09:11.120118 28189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:11.131072 28189 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:11.131963 28189 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:11.135895 28189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7: Generated new TSK 0
I20260805 23:09:11.136624 28189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:09:11.167097 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37667
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:11.284013 28207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:11.284317 28207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:11.284402 28207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:11.288556 28207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:11.288633 28207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:11.288717 28207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:09:11.293953 28207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:11.295003 28207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:11.296025 28207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:11.305641 28214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:11.305660 28213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:11.305739 28207 server_base.cc:1034] running on GCE node
W20260805 23:09:11.305661 28216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:11.306278 28207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:11.307022 28207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:11.308202 28207 hybrid_clock.cc:648] HybridClock initialized: now 1785971351308168 us; error 43 us; skew 500 ppm
Aug 05 23:09:11 dist-test-slave-2d19 krb5kdc[28042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:11.311336 28207 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:09:11.312214 28207 webserver.cc:460] Webserver started at http://127.3.12.65:42237/ using document root <none> and password file <none>
I20260805 23:09:11.312814 28207 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:11.312907 28207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:11.313143 28207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:11.315140 28207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "85eb674e00794af88f20d256d494cd2d"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "274997d92b32f639c193434e6da1f6a6"
server_key_iv: "343661aef30bc61415696551d80d2476"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.315706 28207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "85eb674e00794af88f20d256d494cd2d"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "274997d92b32f639c193434e6da1f6a6"
server_key_iv: "343661aef30bc61415696551d80d2476"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.319749 28207 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260805 23:09:11.322202 28225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.323365 28207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:11.323501 28207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "85eb674e00794af88f20d256d494cd2d"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "274997d92b32f639c193434e6da1f6a6"
server_key_iv: "343661aef30bc61415696551d80d2476"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.323626 28207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:11.343806 28207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:11.346975 28207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:11.347204 28207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:11.347896 28207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:11.348873 28207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:11.348943 28207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.349010 28207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:11.349043 28207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.358098 28207 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:34663
I20260805 23:09:11.358141 28384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:34663 every 8 connection(s)
I20260805 23:09:11.358981 28207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:09:11.367739 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 28207
I20260805 23:09:11.367900 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:09:11.368212 3121 external_mini_cluster.cc:1428] Setting key 0d63bdf30118dc13ebb96964478bdc8c
Aug 05 23:09:11 dist-test-slave-2d19 krb5kdc[28042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:11.373523 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.360738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:33767 (local address 127.3.12.126:37667)
0805 23:09:11.360879 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:11.360883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:11.361854 (+ 971us) server_negotiation.cc:407] Connection header received
0805 23:09:11.363182 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:11.363185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:11.363246 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:11.363361 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:11.364579 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.365122 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.366696 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.367000 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.367107 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.369881 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:11.369891 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:11.369894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:11.369916 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:11.371338 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.371863 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.371867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.371869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.371963 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.372370 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.372373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.372375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.372603 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:11.372715 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:11.373223 (+ 508us) server_negotiation.cc:299] Negotiation successful
0805 23:09:11.373350 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:09:11.374382 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.361226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37667 (local address 127.3.12.65:33767)
0805 23:09:11.361695 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:11.361737 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:11.362972 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:11.363538 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:11.363545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:11.363897 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:11.364422 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.364444 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.365253 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.365257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.366552 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.366564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.367174 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.367178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.367330 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.368009 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:11.368032 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:11.369715 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:11.371461 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:11.371467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:11.371483 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:11.371753 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:11.372077 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:11.372080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:11.372082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:11.372225 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:11.372715 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:11.372723 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:11.372971 (+ 248us) client_negotiation.cc:769] Sending connection context
0805 23:09:11.373165 (+ 194us) client_negotiation.cc:240] Negotiation successful
0805 23:09:11.373491 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
I20260805 23:09:11.375829 28386 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37667
I20260805 23:09:11.376125 28386 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:11.376804 28386 heartbeater.cc:511] Master 127.3.12.126:37667 requested a full tablet report, sending...
I20260805 23:09:11.378633 28114 ts_manager.cc:194] Registered new tserver with Master: 85eb674e00794af88f20d256d494cd2d (127.3.12.65:34663)
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:09:11.380218 28114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:33767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:09:11.399628 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37667
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:11.514317 28392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:11.514578 28392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:11.514622 28392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:11.518648 28392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:11.518713 28392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:11.518770 28392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:09:11.524091 28392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:11.525295 28392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:11.526458 28392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:11.536855 28402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:11.536901 28400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:11.537031 28392 server_base.cc:1034] running on GCE node
W20260805 23:09:11.536885 28398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:11.537705 28392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:11.538439 28392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:11.539616 28392 hybrid_clock.cc:648] HybridClock initialized: now 1785971351539579 us; error 51 us; skew 500 ppm
Aug 05 23:09:11 dist-test-slave-2d19 krb5kdc[28042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:11.542629 28392 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:09:11.543520 28392 webserver.cc:460] Webserver started at http://127.3.12.66:44777/ using document root <none> and password file <none>
I20260805 23:09:11.544049 28392 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:11.544116 28392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:11.544358 28392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:11.546388 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "97f25e849e5a44fba4a397179d059fb4"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "ee7a89318dbce02b22fcb3f576483635"
server_key_iv: "65a54c51ba7d91a8684dbb4b45a74d60"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.547214 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "97f25e849e5a44fba4a397179d059fb4"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "ee7a89318dbce02b22fcb3f576483635"
server_key_iv: "65a54c51ba7d91a8684dbb4b45a74d60"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.551502 28392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260805 23:09:11.554128 28414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.555399 28392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:11.555500 28392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "97f25e849e5a44fba4a397179d059fb4"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "ee7a89318dbce02b22fcb3f576483635"
server_key_iv: "65a54c51ba7d91a8684dbb4b45a74d60"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.555588 28392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:11.568718 28392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:11.571817 28392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:11.571990 28392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:11.572593 28392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:11.573382 28392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:11.573431 28392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.573469 28392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:11.573485 28392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.583184 28392 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:38901
I20260805 23:09:11.583207 28562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:38901 every 8 connection(s)
I20260805 23:09:11.584184 28392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:09:11.587110 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 28392
I20260805 23:09:11.587252 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:09:11.587594 3121 external_mini_cluster.cc:1428] Setting key c450a31ba796ca0108d699df5c621c1f
Aug 05 23:09:11 dist-test-slave-2d19 krb5kdc[28042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:09:11.601312 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.586104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:60019 (local address 127.3.12.126:37667)
0805 23:09:11.586244 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:11.586248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:11.587236 (+ 988us) server_negotiation.cc:407] Connection header received
0805 23:09:11.588880 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:11.588883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:11.588937 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:11.589047 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:11.590651 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.591167 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.592552 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.592949 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.593090 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.596427 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:11.596433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:11.596438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:11.596479 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:11.598412 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.599205 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.599211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.599214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.599322 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.599908 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.599925 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.599927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.600211 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:11.600343 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:11.600965 (+ 622us) server_negotiation.cc:299] Negotiation successful
0805 23:09:11.601120 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:09:11.602459 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.586507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37667 (local address 127.3.12.66:60019)
0805 23:09:11.587055 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:11.587120 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:11.588674 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:11.589240 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:11.589248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:11.589772 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:11.590474 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.590510 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.591300 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.591307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.592426 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.592436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.593116 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.593119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.593271 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.593929 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:11.593946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:11.596127 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:11.598581 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:11.598589 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:11.598609 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:11.599005 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:11.599527 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:11.599546 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:11.599550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:11.599761 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:11.600352 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:11.600377 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:11.600684 (+ 307us) client_negotiation.cc:769] Sending connection context
0805 23:09:11.600848 (+ 164us) client_negotiation.cc:240] Negotiation successful
0805 23:09:11.601328 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
I20260805 23:09:11.604707 28563 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37667
I20260805 23:09:11.604976 28563 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:11.605635 28563 heartbeater.cc:511] Master 127.3.12.126:37667 requested a full tablet report, sending...
I20260805 23:09:11.606868 28114 ts_manager.cc:194] Registered new tserver with Master: 97f25e849e5a44fba4a397179d059fb4 (127.3.12.66:38901)
I20260805 23:09:11.607662 28114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:60019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:09:11.623509 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37667
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:11.738030 28569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:11.738271 28569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:11.738313 28569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:11.742354 28569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:11.742410 28569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:11.742466 28569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:09:11.747629 28569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:11.748862 28569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:11.750135 28569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:11.759634 28586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:11.759647 28580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:11.759647 28578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:11.760459 28569 server_base.cc:1034] running on GCE node
I20260805 23:09:11.760891 28569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:11.761571 28569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:11.762764 28569 hybrid_clock.cc:648] HybridClock initialized: now 1785971351762748 us; error 30 us; skew 500 ppm
Aug 05 23:09:11 dist-test-slave-2d19 krb5kdc[28042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:11.765636 28569 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:09:11.766556 28569 webserver.cc:460] Webserver started at http://127.3.12.67:33555/ using document root <none> and password file <none>
I20260805 23:09:11.767176 28569 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:11.767269 28569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:11.767498 28569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:11.769580 28569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "b780b9f87f6d4423a106abcca5e6a5d5"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "e320773ee798b5f2ce283c8fcb6fe5c1"
server_key_iv: "cb1a762a6b090ed7b51dd79e63783206"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.770179 28569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "b780b9f87f6d4423a106abcca5e6a5d5"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "e320773ee798b5f2ce283c8fcb6fe5c1"
server_key_iv: "cb1a762a6b090ed7b51dd79e63783206"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.773896 28569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260805 23:09:11.776391 28596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.777593 28569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:11.777731 28569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "b780b9f87f6d4423a106abcca5e6a5d5"
format_stamp: "Formatted at 2026-08-05 23:09:11 on dist-test-slave-2d19"
server_key: "e320773ee798b5f2ce283c8fcb6fe5c1"
server_key_iv: "cb1a762a6b090ed7b51dd79e63783206"
server_key_version: "encryptionkey@0"
I20260805 23:09:11.777839 28569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:11.790607 28569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:11.793483 28569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:11.793689 28569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:11.794299 28569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:11.795118 28569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:11.795189 28569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.795262 28569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:11.795306 28569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:11.804390 28569 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:45497
I20260805 23:09:11.804414 28766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:45497 every 8 connection(s)
I20260805 23:09:11.805287 28569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:09:11.810748 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 28569
I20260805 23:09:11.810850 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:09:11.811167 3121 external_mini_cluster.cc:1428] Setting key c90a5d14cdb29fd8e40216a5e145cfeb
Aug 05 23:09:11 dist-test-slave-2d19 krb5kdc[28042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971351, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:11.819290 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.807110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:42671 (local address 127.3.12.126:37667)
0805 23:09:11.807288 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:11.807293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:11.808130 (+ 837us) server_negotiation.cc:407] Connection header received
0805 23:09:11.809679 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:11.809682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:11.809732 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:11.809814 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:11.811183 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.811810 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.813495 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.813723 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.813803 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.816239 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:11.816242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:11.816244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:11.816323 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:11.817524 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.817973 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.817976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.817977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.818078 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.818371 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.818373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.818374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.818546 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:11.818622 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:11.819076 (+ 454us) server_negotiation.cc:299] Negotiation successful
0805 23:09:11.819171 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260805 23:09:11.820190 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.807476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:37667 (local address 127.3.12.67:42671)
0805 23:09:11.807967 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:11.808020 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:11.809476 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:11.809976 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:11.809983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:11.810451 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:11.811028 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.811045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.811934 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.811938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.813365 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.813376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.813853 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.813856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.813990 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.814633 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:11.814650 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:11.816095 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:11.817628 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:11.817632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:11.817644 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:11.817886 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:11.818162 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:11.818164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:11.818165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:11.818292 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:11.818634 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:11.818638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:11.818852 (+ 214us) client_negotiation.cc:769] Sending connection context
0805 23:09:11.819003 (+ 151us) client_negotiation.cc:240] Negotiation successful
0805 23:09:11.819255 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":144,"threads_started":1}
I20260805 23:09:11.821324 28768 heartbeater.cc:347] Connected to a master server at 127.3.12.126:37667
I20260805 23:09:11.821549 28768 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:11.822038 28768 heartbeater.cc:511] Master 127.3.12.126:37667 requested a full tablet report, sending...
I20260805 23:09:11.823045 28114 ts_manager.cc:194] Registered new tserver with Master: b780b9f87f6d4423a106abcca5e6a5d5 (127.3.12.67:45497)
I20260805 23:09:11.823588 28114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:42671
I20260805 23:09:11.825182 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:09:11.834791 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.827226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.3.12.126:37667)
0805 23:09:11.827379 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:11.827384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:11.827594 (+ 210us) server_negotiation.cc:407] Connection header received
0805 23:09:11.827715 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:11.827717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:11.827757 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:11.827821 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:11.828569 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.829094 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.829987 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.830221 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.830302 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.831701 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:11.831704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:11.831706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:11.831729 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:11.833261 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.833650 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.833653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.833654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.833728 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:11.833959 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:11.833961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:11.833963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:11.834193 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:11.834293 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:11.834498 (+ 205us) server_negotiation.cc:299] Negotiation successful
0805 23:09:11.834595 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 23:09:11.837670 28114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:09:11.840754 28114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:09:11.855180 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.849065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:34663 (local address 127.0.0.1:56788)
0805 23:09:11.849719 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:11.849771 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:11.849917 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:11.850712 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:11.850716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:11.850735 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:11.850901 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.850909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.852188 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.852194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.853456 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.853466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.854616 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.854619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.854732 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.854774 (+ 42us) client_negotiation.cc:769] Sending connection context
0805 23:09:11.854889 (+ 115us) client_negotiation.cc:240] Negotiation successful
0805 23:09:11.855006 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"spinlock_wait_cycles":4352,"thread_start_us":91,"threads_started":1}
I20260805 23:09:11.855223 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.849214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.3.12.65:34663)
0805 23:09:11.850206 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:11.850223 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:11.850247 (+ 24us) server_negotiation.cc:407] Connection header received
0805 23:09:11.850319 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:11.850324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:11.850512 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:11.851066 (+ 554us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:11.851242 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.852059 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.853566 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.854502 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.854609 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.854758 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:11.854942 (+ 184us) server_negotiation.cc:299] Negotiation successful
0805 23:09:11.855053 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":110,"threads_started":1}
I20260805 23:09:11.855443 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.849065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38901 (local address 127.0.0.1:39730)
0805 23:09:11.849766 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:11.849778 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:11.849917 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:11.850966 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:11.850970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:11.850997 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:11.851108 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.851120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.852810 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.852814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.853919 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.853928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.855040 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.855044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.855136 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.855159 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:09:11.855210 (+ 51us) client_negotiation.cc:240] Negotiation successful
0805 23:09:11.855289 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":159,"spinlock_wait_cycles":6656,"thread_start_us":101,"threads_started":1}
I20260805 23:09:11.856173 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.849307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.3.12.66:38901)
0805 23:09:11.849736 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:11.849751 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:11.849885 (+ 134us) server_negotiation.cc:407] Connection header received
0805 23:09:11.850094 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:11.850102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:11.850335 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:11.851249 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:11.851406 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.852656 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.854053 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.854889 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.855711 (+ 822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.855852 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:11.855902 (+ 50us) server_negotiation.cc:299] Negotiation successful
0805 23:09:11.855988 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":123,"threads_started":1}
I20260805 23:09:11.856344 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.849873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:45497 (local address 127.0.0.1:50554)
0805 23:09:11.850487 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:11.850501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:11.850589 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:11.851208 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:11.851213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:11.851231 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:11.851311 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.851316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.853601 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.853604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.854450 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:11.854458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.855916 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.855920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:11.856003 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.856024 (+ 21us) client_negotiation.cc:769] Sending connection context
0805 23:09:11.856106 (+ 82us) client_negotiation.cc:240] Negotiation successful
0805 23:09:11.856176 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":119,"threads_started":1}
I20260805 23:09:11.856442 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:11.850315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.3.12.67:45497)
0805 23:09:11.850763 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:11.850780 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:11.850802 (+ 22us) server_negotiation.cc:407] Connection header received
0805 23:09:11.850880 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:11.850888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:11.851065 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:11.851223 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:11.851406 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.852658 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.854578 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:11.855630 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:11.855728 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:11.855852 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:11.856167 (+ 315us) server_negotiation.cc:299] Negotiation successful
0805 23:09:11.856290 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":103,"threads_started":1}
I20260805 23:09:11.858608 28289 tablet_service.cc:1467] Processing CreateTablet for tablet e179a0838b034edaa2fd846afe2a4e13 (DEFAULT_TABLE table=test-table [id=fd7670c6069c4714853ecb9ef3d20319]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:11.859112 28665 tablet_service.cc:1467] Processing CreateTablet for tablet e179a0838b034edaa2fd846afe2a4e13 (DEFAULT_TABLE table=test-table [id=fd7670c6069c4714853ecb9ef3d20319]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:11.859140 28472 tablet_service.cc:1467] Processing CreateTablet for tablet e179a0838b034edaa2fd846afe2a4e13 (DEFAULT_TABLE table=test-table [id=fd7670c6069c4714853ecb9ef3d20319]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:11.859807 28289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e179a0838b034edaa2fd846afe2a4e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:11.860121 28665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e179a0838b034edaa2fd846afe2a4e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:11.860215 28472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e179a0838b034edaa2fd846afe2a4e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:11.867168 28795 tablet_bootstrap.cc:492] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Bootstrap starting.
I20260805 23:09:11.868621 28797 tablet_bootstrap.cc:492] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Bootstrap starting.
I20260805 23:09:11.870074 28795 tablet_bootstrap.cc:654] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:11.870890 28797 tablet_bootstrap.cc:654] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:11.870992 28795 log.cc:824] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:11.871425 28796 tablet_bootstrap.cc:492] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d: Bootstrap starting.
I20260805 23:09:11.872215 28797 log.cc:824] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:11.873618 28795 tablet_bootstrap.cc:492] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: No bootstrap required, opened a new log
I20260805 23:09:11.873801 28796 tablet_bootstrap.cc:654] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:11.873834 28795 ts_tablet_manager.cc:1397] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260805 23:09:11.874866 28796 log.cc:824] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:11.876813 28796 tablet_bootstrap.cc:492] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d: No bootstrap required, opened a new log
I20260805 23:09:11.877036 28796 ts_tablet_manager.cc:1397] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:09:11.877593 28795 raft_consensus.cc:348] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
I20260805 23:09:11.877900 28795 raft_consensus.cc:374] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:11.877974 28795 raft_consensus.cc:729] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f25e849e5a44fba4a397179d059fb4, State: Initialized, Role: FOLLOWER
I20260805 23:09:11.878468 28795 consensus_queue.cc:260] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
I20260805 23:09:11.879160 28563 heartbeater.cc:503] Master 127.3.12.126:37667 was elected leader, sending a full tablet report...
I20260805 23:09:11.879529 28795 ts_tablet_manager.cc:1428] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260805 23:09:11.881582 28797 tablet_bootstrap.cc:492] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: No bootstrap required, opened a new log
I20260805 23:09:11.881805 28797 ts_tablet_manager.cc:1397] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260805 23:09:11.882059 28796 raft_consensus.cc:348] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
I20260805 23:09:11.882326 28796 raft_consensus.cc:374] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:11.882407 28796 raft_consensus.cc:729] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85eb674e00794af88f20d256d494cd2d, State: Initialized, Role: FOLLOWER
I20260805 23:09:11.882953 28796 consensus_queue.cc:260] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
I20260805 23:09:11.883677 28386 heartbeater.cc:503] Master 127.3.12.126:37667 was elected leader, sending a full tablet report...
I20260805 23:09:11.884068 28796 ts_tablet_manager.cc:1428] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260805 23:09:11.885533 28797 raft_consensus.cc:348] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
I20260805 23:09:11.885792 28797 raft_consensus.cc:374] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:11.885864 28797 raft_consensus.cc:729] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b780b9f87f6d4423a106abcca5e6a5d5, State: Initialized, Role: FOLLOWER
I20260805 23:09:11.886313 28797 consensus_queue.cc:260] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
I20260805 23:09:11.886956 28768 heartbeater.cc:503] Master 127.3.12.126:37667 was elected leader, sending a full tablet report...
I20260805 23:09:11.887367 28797 ts_tablet_manager.cc:1428] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260805 23:09:12.041661 28805 raft_consensus.cc:482] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:09:12.041903 28805 raft_consensus.cc:504] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
I20260805 23:09:12.043167 28805 leader_election.cc:288] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f25e849e5a44fba4a397179d059fb4 (127.3.12.66:38901), b780b9f87f6d4423a106abcca5e6a5d5 (127.3.12.67:45497)
I20260805 23:09:12.047822 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.043414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38901 (local address 127.3.12.65:35963)
0805 23:09:12.044005 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:12.044019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:12.044124 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:12.044439 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:12.044441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:12.044457 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:12.044612 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:12.044618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.045329 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.045332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:12.046414 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:12.046427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.047332 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.047338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:12.047429 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.047455 (+ 26us) client_negotiation.cc:769] Sending connection context
0805 23:09:12.047532 (+ 77us) client_negotiation.cc:240] Negotiation successful
0805 23:09:12.047607 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":73,"thread_start_us":168,"threads_started":1}
I20260805 23:09:12.047988 28807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.043406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:45497 (local address 127.3.12.65:55299)
0805 23:09:12.043831 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:12.043844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:12.043946 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:12.044216 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:12.044220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:12.044237 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:12.044421 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:12.044429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.045254 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.045260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:12.046429 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:12.046445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.047512 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.047518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:12.047616 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.047634 (+ 18us) client_negotiation.cc:769] Sending connection context
0805 23:09:12.047759 (+ 125us) client_negotiation.cc:240] Negotiation successful
0805 23:09:12.047826 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":86,"thread_start_us":168,"threads_started":1}
I20260805 23:09:12.047977 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.043563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:55299 (local address 127.3.12.67:45497)
0805 23:09:12.043695 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:12.043702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:12.043941 (+ 239us) server_negotiation.cc:407] Connection header received
0805 23:09:12.044038 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:12.044044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:12.044105 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:12.044187 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:12.044542 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.045139 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.046599 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.047276 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.047380 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.047441 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:12.047758 (+ 317us) server_negotiation.cc:299] Negotiation successful
0805 23:09:12.047847 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:09:12.048535 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.043521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:35963 (local address 127.3.12.66:38901)
0805 23:09:12.043669 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:12.043674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:12.044115 (+ 441us) server_negotiation.cc:407] Connection header received
0805 23:09:12.044246 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:12.044275 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:12.044327 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:12.044411 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:12.044722 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.045212 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.046538 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.047125 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.047237 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.047313 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:12.048311 (+ 998us) server_negotiation.cc:299] Negotiation successful
0805 23:09:12.048424 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:09:12.049271 28697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e179a0838b034edaa2fd846afe2a4e13" candidate_uuid: "85eb674e00794af88f20d256d494cd2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" is_pre_election: true
I20260805 23:09:12.049271 28502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e179a0838b034edaa2fd846afe2a4e13" candidate_uuid: "85eb674e00794af88f20d256d494cd2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f25e849e5a44fba4a397179d059fb4" is_pre_election: true
I20260805 23:09:12.049592 28502 raft_consensus.cc:2454] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85eb674e00794af88f20d256d494cd2d in term 0.
I20260805 23:09:12.049598 28697 raft_consensus.cc:2454] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85eb674e00794af88f20d256d494cd2d in term 0.
I20260805 23:09:12.050079 28228 leader_election.cc:302] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85eb674e00794af88f20d256d494cd2d, 97f25e849e5a44fba4a397179d059fb4; no voters:
I20260805 23:09:12.050384 28805 raft_consensus.cc:2781] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:09:12.050473 28805 raft_consensus.cc:482] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:09:12.050516 28805 raft_consensus.cc:3037] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:12.051575 28805 raft_consensus.cc:504] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
I20260805 23:09:12.051951 28805 leader_election.cc:288] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [CANDIDATE]: Term 1 election: Requested vote from peers 97f25e849e5a44fba4a397179d059fb4 (127.3.12.66:38901), b780b9f87f6d4423a106abcca5e6a5d5 (127.3.12.67:45497)
I20260805 23:09:12.052345 28502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e179a0838b034edaa2fd846afe2a4e13" candidate_uuid: "85eb674e00794af88f20d256d494cd2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f25e849e5a44fba4a397179d059fb4"
I20260805 23:09:12.052376 28697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e179a0838b034edaa2fd846afe2a4e13" candidate_uuid: "85eb674e00794af88f20d256d494cd2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b780b9f87f6d4423a106abcca5e6a5d5"
I20260805 23:09:12.052485 28502 raft_consensus.cc:3037] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:12.052511 28697 raft_consensus.cc:3037] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:12.053544 28502 raft_consensus.cc:2454] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85eb674e00794af88f20d256d494cd2d in term 1.
I20260805 23:09:12.053643 28697 raft_consensus.cc:2454] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85eb674e00794af88f20d256d494cd2d in term 1.
I20260805 23:09:12.053903 28228 leader_election.cc:302] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85eb674e00794af88f20d256d494cd2d, 97f25e849e5a44fba4a397179d059fb4; no voters:
I20260805 23:09:12.054067 28805 raft_consensus.cc:2781] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:12.054309 28805 raft_consensus.cc:686] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [term 1 LEADER]: Becoming Leader. State: Replica: 85eb674e00794af88f20d256d494cd2d, State: Running, Role: LEADER
I20260805 23:09:12.054663 28805 consensus_queue.cc:237] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } }
W20260805 23:09:12.056296 28769 tablet.cc:2369] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:09:12.058079 28113 catalog_manager.cc:5495] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 85eb674e00794af88f20d256d494cd2d (127.3.12.65). New cstate: current_term: 1 leader_uuid: "85eb674e00794af88f20d256d494cd2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85eb674e00794af88f20d256d494cd2d" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 } health_report { overall_health: UNKNOWN } } }
W20260805 23:09:12.085471 28564 tablet.cc:2369] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:09:12.093242 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.089573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.3.12.65:34663)
0805 23:09:12.089732 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:12.089737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:12.089752 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:09:12.089887 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:12.089890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:12.089946 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:12.090069 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:12.090328 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.090833 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.091715 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.091949 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.092074 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.092430 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:12.092825 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:12.092960 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:12.093024 (+ 64us) server_negotiation.cc:299] Negotiation successful
0805 23:09:12.093092 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 23:09:12.099948 28697 raft_consensus.cc:1261] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 1 FOLLOWER]: Refusing update from remote peer 85eb674e00794af88f20d256d494cd2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:12.099949 28502 raft_consensus.cc:1261] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 1 FOLLOWER]: Refusing update from remote peer 85eb674e00794af88f20d256d494cd2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:12.100771 28805 consensus_queue.cc:1035] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b780b9f87f6d4423a106abcca5e6a5d5" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 45497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:12.101019 28810 consensus_queue.cc:1035] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f25e849e5a44fba4a397179d059fb4" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:12.109206 28817 mvcc.cc:205] Tried to move back new op lower bound from 7315338658193002496 to 7315338658018709504. Current Snapshot: MvccSnapshot[applied={T|T < 7315338658193002496}]
I20260805 23:09:12.110657 28815 mvcc.cc:205] Tried to move back new op lower bound from 7315338658193002496 to 7315338658018709504. Current Snapshot: MvccSnapshot[applied={T|T < 7315338658193002496}]
I20260805 23:09:12.118414 28113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:12.118616 28113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:12.120980 28113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8bca580adfb4a84b07cd2fbdda04fc7: Sending DeleteTablet for 3 replicas of tablet e179a0838b034edaa2fd846afe2a4e13
I20260805 23:09:12.121790 28472 tablet_service.cc:1514] Processing DeleteTablet for tablet e179a0838b034edaa2fd846afe2a4e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:12 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:39730
I20260805 23:09:12.121851 28289 tablet_service.cc:1514] Processing DeleteTablet for tablet e179a0838b034edaa2fd846afe2a4e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:12 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:56788
I20260805 23:09:12.121898 28665 tablet_service.cc:1514] Processing DeleteTablet for tablet e179a0838b034edaa2fd846afe2a4e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:12 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:50554
I20260805 23:09:12.122319 28827 tablet_replica.cc:330] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: stopping tablet replica
I20260805 23:09:12.122396 28829 tablet_replica.cc:330] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: stopping tablet replica
I20260805 23:09:12.122638 28829 raft_consensus.cc:2229] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:12.122638 28827 raft_consensus.cc:2229] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:12.122937 28827 raft_consensus.cc:2258] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:12.122937 28829 raft_consensus.cc:2258] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:12.123435 28828 tablet_replica.cc:330] T e179a0838b034edaa2fd846afe2a4e13 P 85eb674e00794af88f20d256d494cd2d: stopping tablet replica
I20260805 23:09:12.123548 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 28207
I20260805 23:09:12.129236 28829 ts_tablet_manager.cc:1905] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:12.132732 28827 ts_tablet_manager.cc:1905] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:12.133167 28829 ts_tablet_manager.cc:1918] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:12.133255 28829 log.cc:1196] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/e179a0838b034edaa2fd846afe2a4e13
I20260805 23:09:12.133829 28829 ts_tablet_manager.cc:1939] T e179a0838b034edaa2fd846afe2a4e13 P b780b9f87f6d4423a106abcca5e6a5d5: Deleting consensus metadata
I20260805 23:09:12.135295 28097 catalog_manager.cc:4841] TS b780b9f87f6d4423a106abcca5e6a5d5 (127.3.12.67:45497): tablet e179a0838b034edaa2fd846afe2a4e13 (table test-table [id=fd7670c6069c4714853ecb9ef3d20319]) successfully deleted
I20260805 23:09:12.135394 28827 ts_tablet_manager.cc:1918] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:12.135527 28827 log.cc:1196] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/e179a0838b034edaa2fd846afe2a4e13
I20260805 23:09:12.135808 28827 ts_tablet_manager.cc:1939] T e179a0838b034edaa2fd846afe2a4e13 P 97f25e849e5a44fba4a397179d059fb4: Deleting consensus metadata
I20260805 23:09:12.136960 28091 catalog_manager.cc:4841] TS 97f25e849e5a44fba4a397179d059fb4 (127.3.12.66:38901): tablet e179a0838b034edaa2fd846afe2a4e13 (table test-table [id=fd7670c6069c4714853ecb9ef3d20319]) successfully deleted
W20260805 23:09:12.140189 28097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:34663 (error 108)
I20260805 23:09:12.140368 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 28392
I20260805 23:09:12.146320 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.145892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:34663 (local address 127.0.0.1:56804)
0805 23:09:12.146083 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:12.146193 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:34663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260805 23:09:12.146569 28097 catalog_manager.cc:4584] TS 85eb674e00794af88f20d256d494cd2d (127.3.12.65:34663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e179a0838b034edaa2fd846afe2a4e13: Network error: Client connection negotiation failed: client connection to 127.3.12.65:34663: connect: Connection refused (error 111)
I20260805 23:09:12.153844 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 28569
I20260805 23:09:12.166136 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 28067
2026-08-05T23:09:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2651 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-05T23:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:12Z Disabled control of system clock
I20260805 23:09:12.203289 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42957
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 23:09:12.318243 28837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:12.318624 28837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:12.318704 28837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:12.322710 28837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:12.322767 28837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:12.322782 28837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:12.322799 28837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:12.322858 28837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:12.327540 28837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:42957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:12.328752 28837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:12.330137 28837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:12.338599 28846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:12.338650 28850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:12.338555 28844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:12.338742 28837 server_base.cc:1034] running on GCE node
I20260805 23:09:12.339378 28837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:12.340584 28837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:12.341753 28837 hybrid_clock.cc:648] HybridClock initialized: now 1785971352341717 us; error 46 us; skew 500 ppm
I20260805 23:09:12.343648 28837 webserver.cc:460] Webserver started at http://127.3.12.126:33961/ using document root <none> and password file <none>
I20260805 23:09:12.344230 28837 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:12.344359 28837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:12.344621 28837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:12.346921 28837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "909e998ede6b43518fd962a5cc493953"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "2b70c009d4125e9bb383fd493d307d96"
server_key_iv: "7d60fdd62d38fba9eda3de5e5b5917fe"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.347659 28837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "909e998ede6b43518fd962a5cc493953"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "2b70c009d4125e9bb383fd493d307d96"
server_key_iv: "7d60fdd62d38fba9eda3de5e5b5917fe"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.351429 28837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260805 23:09:12.353873 28857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:12.355115 28837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:09:12.355244 28837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "909e998ede6b43518fd962a5cc493953"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "2b70c009d4125e9bb383fd493d307d96"
server_key_iv: "7d60fdd62d38fba9eda3de5e5b5917fe"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.355352 28837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:12.372434 28837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:12.373162 28837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:12.373374 28837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:12.380524 28837 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:42957
I20260805 23:09:12.380528 28923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:42957 every 8 connection(s)
I20260805 23:09:12.381469 28837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:12.384577 28924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:12.389705 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 28837
I20260805 23:09:12.389847 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:12.390161 3121 external_mini_cluster.cc:1428] Setting key 015aea23fe3874b199a9d763171a57bc
I20260805 23:09:12.390400 28924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953: Bootstrap starting.
I20260805 23:09:12.393101 28924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:12.393922 28924 log.cc:824] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:12.395859 28924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953: No bootstrap required, opened a new log
I20260805 23:09:12.399709 28924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e998ede6b43518fd962a5cc493953" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42957 } }
I20260805 23:09:12.399921 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.391676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.3.12.126:42957)
0805 23:09:12.392443 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:12.392459 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:12.392503 (+ 44us) server_negotiation.cc:407] Connection header received
0805 23:09:12.393133 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:12.393149 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:12.393623 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:12.393964 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:12.394397 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.395405 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.396411 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.396888 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.397065 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.398109 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:12.398115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:12.398128 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:12.398551 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:12.398611 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:12.398633 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:12.398755 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:12.398930 (+ 175us) server_negotiation.cc:299] Negotiation successful
0805 23:09:12.399194 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":163,"threads_started":1}
I20260805 23:09:12.399946 28924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:12.400072 28924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909e998ede6b43518fd962a5cc493953, State: Initialized, Role: FOLLOWER
I20260805 23:09:12.400559 28924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e998ede6b43518fd962a5cc493953" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42957 } }
I20260805 23:09:12.400713 28924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:12.400789 28924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:12.400894 28924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:12.401790 28924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e998ede6b43518fd962a5cc493953" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42957 } }
I20260805 23:09:12.402130 28924 leader_election.cc:302] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 909e998ede6b43518fd962a5cc493953; no voters:
I20260805 23:09:12.402444 28924 leader_election.cc:288] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:12.402639 28933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:12.403020 28933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [term 1 LEADER]: Becoming Leader. State: Replica: 909e998ede6b43518fd962a5cc493953, State: Running, Role: LEADER
I20260805 23:09:12.403393 28933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e998ede6b43518fd962a5cc493953" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42957 } }
I20260805 23:09:12.403705 28924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:12.405133 28935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 909e998ede6b43518fd962a5cc493953. Latest consensus state: current_term: 1 leader_uuid: "909e998ede6b43518fd962a5cc493953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e998ede6b43518fd962a5cc493953" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42957 } } }
I20260805 23:09:12.405272 28935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:12.405598 28934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "909e998ede6b43518fd962a5cc493953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909e998ede6b43518fd962a5cc493953" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42957 } } }
I20260805 23:09:12.405699 28934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:12.405843 28948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:12.408905 28948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:12.414368 28948 catalog_manager.cc:1347] Generated new cluster ID: 40ffec5a3f61497f947c9a1b8e4b75ee
I20260805 23:09:12.414444 28948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:12.438542 28948 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:12.439846 28948 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:12.444321 28948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953: Generated new TSK 0
I20260805 23:09:12.445173 28948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:12.451798 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42957
--builtin_ntp_servers=127.3.12.84:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 23:09:12.568672 28957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:12.568997 28957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:12.569073 28957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:12.573180 28957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:12.573268 28957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:12.573361 28957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:09:12.578346 28957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:12.579573 28957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:12.581120 28957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:12.591236 28965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:12.591336 28957 server_base.cc:1034] running on GCE node
W20260805 23:09:12.591286 28963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:12.591244 28962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:12.592017 28957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:12.592747 28957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:12.593947 28957 hybrid_clock.cc:648] HybridClock initialized: now 1785971352593924 us; error 33 us; skew 500 ppm
I20260805 23:09:12.595913 28957 webserver.cc:460] Webserver started at http://127.3.12.65:34797/ using document root <none> and password file <none>
I20260805 23:09:12.596653 28957 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:12.596750 28957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:12.596997 28957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:12.599162 28957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "30002f680f8f4632838b2d94de0c0920"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "afdf82f5c57f7bcc733af9f52ef39a06"
server_key_iv: "f4acea11167ffa7deda96ed38769db8a"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.599753 28957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "30002f680f8f4632838b2d94de0c0920"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "afdf82f5c57f7bcc733af9f52ef39a06"
server_key_iv: "f4acea11167ffa7deda96ed38769db8a"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.603684 28957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:09:12.606498 28972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:12.607774 28957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260805 23:09:12.607921 28957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "30002f680f8f4632838b2d94de0c0920"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "afdf82f5c57f7bcc733af9f52ef39a06"
server_key_iv: "f4acea11167ffa7deda96ed38769db8a"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.608074 28957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:12.634747 28957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:12.635535 28957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:12.635741 28957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:12.636400 28957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:12.637388 28957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:12.637459 28957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:12.637533 28957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:12.637578 28957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:12.646867 28957 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:41405
I20260805 23:09:12.646895 29131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:41405 every 8 connection(s)
I20260805 23:09:12.647806 28957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:09:12.649823 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 28957
I20260805 23:09:12.649963 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:09:12.650252 3121 external_mini_cluster.cc:1428] Setting key 85f5a8dfef5551e65910d3df04d9b02c
I20260805 23:09:12.652367 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42957
--builtin_ntp_servers=127.3.12.84:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 23:09:12.658624 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.649803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:42411 (local address 127.3.12.126:42957)
0805 23:09:12.649989 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:12.649994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:12.651183 (+ 1189us) server_negotiation.cc:407] Connection header received
0805 23:09:12.652758 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:12.652764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:12.652824 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:12.652919 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:12.653832 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.654577 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.656182 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.656493 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.656644 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.657830 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:12.657834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:12.657837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:12.657851 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:12.657879 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:12.657887 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:12.658035 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:12.658379 (+ 344us) server_negotiation.cc:299] Negotiation successful
0805 23:09:12.658453 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 23:09:12.659605 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.650246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42957 (local address 127.3.12.65:42411)
0805 23:09:12.651016 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:12.651072 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:12.652497 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:12.653095 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:12.653102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:12.653187 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:12.653685 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:12.653698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.654747 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.654753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:12.656039 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:12.656052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.656692 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.656696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:12.656868 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.657553 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:12.657554 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:12.657597 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:12.657664 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:12.658035 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:12.658048 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:12.658056 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 23:09:12.658378 (+ 322us) client_negotiation.cc:240] Negotiation successful
0805 23:09:12.658698 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":146,"threads_started":1}
I20260805 23:09:12.661007 29133 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42957
I20260805 23:09:12.661291 29133 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:12.661866 29133 heartbeater.cc:511] Master 127.3.12.126:42957 requested a full tablet report, sending...
I20260805 23:09:12.663668 28876 ts_manager.cc:194] Registered new tserver with Master: 30002f680f8f4632838b2d94de0c0920 (127.3.12.65:41405)
I20260805 23:09:12.665505 28876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.65:42411
W20260805 23:09:12.770046 29139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:12.770310 29139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:12.770349 29139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:12.774283 29139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:12.774335 29139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:12.774387 29139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:09:12.779217 29139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:12.780231 29139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:12.781611 29139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:12.791781 29154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:12.791848 29139 server_base.cc:1034] running on GCE node
W20260805 23:09:12.791790 29152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:12.791790 29156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:12.792563 29139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:12.793386 29139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:12.794579 29139 hybrid_clock.cc:648] HybridClock initialized: now 1785971352794561 us; error 29 us; skew 500 ppm
I20260805 23:09:12.796610 29139 webserver.cc:460] Webserver started at http://127.3.12.66:40671/ using document root <none> and password file <none>
I20260805 23:09:12.797204 29139 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:12.797298 29139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:12.797540 29139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:12.799608 29139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "535638e06b71070c37c4daec42a068e3"
server_key_iv: "d8a276ca1ab2789e1d670508c565e667"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.800189 29139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "535638e06b71070c37c4daec42a068e3"
server_key_iv: "d8a276ca1ab2789e1d670508c565e667"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.803884 29139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:09:12.806479 29164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:12.807648 29139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260805 23:09:12.807789 29139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "535638e06b71070c37c4daec42a068e3"
server_key_iv: "d8a276ca1ab2789e1d670508c565e667"
server_key_version: "encryptionkey@0"
I20260805 23:09:12.807904 29139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:12.833505 29139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:12.834237 29139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:12.834446 29139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:12.835090 29139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:12.836061 29139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:12.836133 29139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:12.836207 29139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:12.836252 29139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:12.845387 29139 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:38677
I20260805 23:09:12.845417 29319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:38677 every 8 connection(s)
I20260805 23:09:12.846324 29139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:09:12.850931 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 29139
I20260805 23:09:12.851073 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:09:12.851380 3121 external_mini_cluster.cc:1428] Setting key 797c12ca415b2d261deef0c6688a42c9
I20260805 23:09:12.853611 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42957
--builtin_ntp_servers=127.3.12.84:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 23:09:12.857211 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.848173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:60957 (local address 127.3.12.126:42957)
0805 23:09:12.848351 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:12.848356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:12.849482 (+ 1126us) server_negotiation.cc:407] Connection header received
0805 23:09:12.851050 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:12.851054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:12.851120 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:12.851240 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:12.852239 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.852894 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.854646 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.855041 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.855137 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.856628 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:12.856634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:12.856640 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:12.856658 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:12.856690 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:12.856701 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:12.856788 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:12.856936 (+ 148us) server_negotiation.cc:299] Negotiation successful
0805 23:09:12.857007 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260805 23:09:12.857980 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:12.848631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42957 (local address 127.3.12.66:60957)
0805 23:09:12.849326 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:12.849369 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:12.850779 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:12.851443 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:12.851453 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:12.851566 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:12.852069 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:12.852107 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.853030 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.853034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:12.854476 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:12.854503 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:12.855350 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:12.855356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:12.855611 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:12.856342 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:12.856344 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:12.856385 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:12.856450 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:12.856794 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:12.856801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:12.856806 (+ 5us) client_negotiation.cc:769] Sending connection context
0805 23:09:12.856902 (+ 96us) client_negotiation.cc:240] Negotiation successful
0805 23:09:12.857133 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
I20260805 23:09:12.859376 29320 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42957
I20260805 23:09:12.859638 29320 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:12.860235 29320 heartbeater.cc:511] Master 127.3.12.126:42957 requested a full tablet report, sending...
I20260805 23:09:12.861371 28876 ts_manager.cc:194] Registered new tserver with Master: 41e8db3cc1f34b4d9fd2196e0b7c9a3d (127.3.12.66:38677)
I20260805 23:09:12.861948 28876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.66:60957
W20260805 23:09:12.971122 29323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:12.971411 29323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:12.971486 29323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:12.975595 29323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:12.975675 29323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:12.975768 29323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:09:12.980655 29323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:12.981812 29323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:12.983237 29323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:12.993055 29332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:12.993046 29334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:12.993046 29331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:12.993130 29323 server_base.cc:1034] running on GCE node
I20260805 23:09:12.993697 29323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:12.994314 29323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:12.995533 29323 hybrid_clock.cc:648] HybridClock initialized: now 1785971352995518 us; error 34 us; skew 500 ppm
I20260805 23:09:12.997382 29323 webserver.cc:460] Webserver started at http://127.3.12.67:45081/ using document root <none> and password file <none>
I20260805 23:09:12.997915 29323 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:12.997975 29323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:12.998142 29323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:13.000101 29323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "31854ac9e71c417cb4848ad6d188cbce"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "ee6689d5a396204f0d24b46fc491ecf7"
server_key_iv: "c299bf1d9bad67cbc8c32a1ed541f2cf"
server_key_version: "encryptionkey@0"
I20260805 23:09:13.000705 29323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "31854ac9e71c417cb4848ad6d188cbce"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "ee6689d5a396204f0d24b46fc491ecf7"
server_key_iv: "c299bf1d9bad67cbc8c32a1ed541f2cf"
server_key_version: "encryptionkey@0"
I20260805 23:09:13.004458 29323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 23:09:13.006945 29340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:13.008173 29323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:13.008329 29323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "31854ac9e71c417cb4848ad6d188cbce"
format_stamp: "Formatted at 2026-08-05 23:09:12 on dist-test-slave-2d19"
server_key: "ee6689d5a396204f0d24b46fc491ecf7"
server_key_iv: "c299bf1d9bad67cbc8c32a1ed541f2cf"
server_key_version: "encryptionkey@0"
I20260805 23:09:13.008424 29323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:13.056772 29323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:13.057482 29323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:13.057648 29323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:13.058215 29323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:13.059252 29323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:13.059304 29323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:13.059345 29323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:13.059361 29323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:13.068495 29323 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:44725
I20260805 23:09:13.068517 29503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:44725 every 8 connection(s)
I20260805 23:09:13.069446 29323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:09:13.072342 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 29323
I20260805 23:09:13.072453 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:09:13.072767 3121 external_mini_cluster.cc:1428] Setting key c44ca3ff89bc0a65270e9e45eebbc6dd
I20260805 23:09:13.079617 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.071396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:35245 (local address 127.3.12.126:42957)
0805 23:09:13.071565 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:13.071570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:13.072769 (+ 1199us) server_negotiation.cc:407] Connection header received
0805 23:09:13.074617 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:13.074620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:13.074666 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:13.074741 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:13.075748 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.076422 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.077582 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.077807 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.077894 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.079132 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:13.079135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:13.079138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:13.079154 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:13.079174 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:13.079183 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:13.079243 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:13.079443 (+ 200us) server_negotiation.cc:299] Negotiation successful
0805 23:09:13.079496 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 23:09:13.080623 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.071786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42957 (local address 127.3.12.67:35245)
0805 23:09:13.072610 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:13.072653 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:13.074387 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:13.074954 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:13.074963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:13.075135 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:13.075623 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.075636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.076544 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.076547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.077460 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.077470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.077998 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.078002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.078151 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.078847 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:13.078849 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:13.078909 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:13.078982 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:13.079292 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:13.079310 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:13.079315 (+ 5us) client_negotiation.cc:769] Sending connection context
0805 23:09:13.079433 (+ 118us) client_negotiation.cc:240] Negotiation successful
0805 23:09:13.079659 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":126,"threads_started":1}
I20260805 23:09:13.081743 29504 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42957
I20260805 23:09:13.081982 29504 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:13.082525 29504 heartbeater.cc:511] Master 127.3.12.126:42957 requested a full tablet report, sending...
I20260805 23:09:13.083832 28876 ts_manager.cc:194] Registered new tserver with Master: 31854ac9e71c417cb4848ad6d188cbce (127.3.12.67:44725)
I20260805 23:09:13.084483 28876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.67:35245
I20260805 23:09:13.086778 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:09:13.092742 28930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.088777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.3.12.126:42957)
0805 23:09:13.088919 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:13.088925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:13.089104 (+ 179us) server_negotiation.cc:407] Connection header received
0805 23:09:13.089245 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:13.089248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:13.089297 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:13.089382 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:13.089685 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.090196 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.091076 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.091295 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.091407 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.092286 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:13.092290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:13.092293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:13.092308 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:13.092331 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:13.092339 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:13.092422 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:13.092545 (+ 123us) server_negotiation.cc:299] Negotiation successful
0805 23:09:13.092605 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260805 23:09:13.095744 28876 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:09:13.098776 28876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:09:13.114205 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.108817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.3.12.67:44725)
0805 23:09:13.109416 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:13.109428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:13.109461 (+ 33us) server_negotiation.cc:407] Connection header received
0805 23:09:13.109537 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:13.109541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:13.109722 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:13.109874 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:13.110096 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.110885 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.112919 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.113529 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.113659 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.113856 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:13.113946 (+ 90us) server_negotiation.cc:299] Negotiation successful
0805 23:09:13.114043 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":185,"threads_started":1}
I20260805 23:09:13.114205 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.108547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:44725 (local address 127.0.0.1:41462)
0805 23:09:13.108805 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:13.108818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:13.108903 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:13.109855 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:13.109858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:13.109876 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:13.109986 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.109993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.111060 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.111063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.112778 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.112788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.113657 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.113661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.113765 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.113802 (+ 37us) client_negotiation.cc:769] Sending connection context
0805 23:09:13.113916 (+ 114us) client_negotiation.cc:240] Negotiation successful
0805 23:09:13.114037 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":37,"spinlock_wait_cycles":8960,"thread_start_us":92,"threads_started":1}
I20260805 23:09:13.114497 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.107906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38677 (local address 127.0.0.1:57374)
0805 23:09:13.108189 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:13.108201 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:13.108305 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:13.109568 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:13.109576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:13.109609 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:13.110000 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.110007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.111471 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.111474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.112705 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.112715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.113870 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.113874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.114045 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.114070 (+ 25us) client_negotiation.cc:769] Sending connection context
0805 23:09:13.114136 (+ 66us) client_negotiation.cc:240] Negotiation successful
0805 23:09:13.114207 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":67,"spinlock_wait_cycles":384,"thread_start_us":98,"threads_started":1}
I20260805 23:09:13.114573 29522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.107518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:41405 (local address 127.0.0.1:49186)
0805 23:09:13.107872 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:13.107918 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:13.108028 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:13.109587 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:13.109589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:13.109614 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:13.109789 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.109798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.111201 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.111204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.112674 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.112687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.113870 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.113874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.114242 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.114262 (+ 20us) client_negotiation.cc:769] Sending connection context
0805 23:09:13.114338 (+ 76us) client_negotiation.cc:240] Negotiation successful
0805 23:09:13.114403 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":329,"spinlock_wait_cycles":5120,"thread_start_us":88,"threads_started":1}
I20260805 23:09:13.115272 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.108389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.3.12.66:38677)
0805 23:09:13.108781 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:13.108793 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:13.108821 (+ 28us) server_negotiation.cc:407] Connection header received
0805 23:09:13.109159 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:13.109165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:13.109381 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:13.109549 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:13.110124 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.111339 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.112924 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.113724 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.114732 (+ 1008us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.114920 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:13.114982 (+ 62us) server_negotiation.cc:299] Negotiation successful
0805 23:09:13.115088 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":149,"threads_started":1}
I20260805 23:09:13.115702 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.107617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.3.12.65:41405)
0805 23:09:13.109045 (+ 1428us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:13.109053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:13.109078 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:09:13.109155 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:13.109161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:13.109408 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:13.109702 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:13.109895 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.111113 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.112813 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.113724 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.114069 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.114253 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:13.115464 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0805 23:09:13.115559 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":80,"threads_started":1}
I20260805 23:09:13.117573 29401 tablet_service.cc:1467] Processing CreateTablet for tablet cc42c549c0274da0b339cb547d8aa6b4 (DEFAULT_TABLE table=test-table [id=398cd21764c745ef967c6a08aacb9a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:13.118455 29401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc42c549c0274da0b339cb547d8aa6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:13.118045 29231 tablet_service.cc:1467] Processing CreateTablet for tablet cc42c549c0274da0b339cb547d8aa6b4 (DEFAULT_TABLE table=test-table [id=398cd21764c745ef967c6a08aacb9a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:13.118595 29038 tablet_service.cc:1467] Processing CreateTablet for tablet cc42c549c0274da0b339cb547d8aa6b4 (DEFAULT_TABLE table=test-table [id=398cd21764c745ef967c6a08aacb9a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:13.119076 29231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc42c549c0274da0b339cb547d8aa6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:13.119521 29038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc42c549c0274da0b339cb547d8aa6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:13.124383 29529 tablet_bootstrap.cc:492] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: Bootstrap starting.
I20260805 23:09:13.126929 29530 tablet_bootstrap.cc:492] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Bootstrap starting.
I20260805 23:09:13.127239 29529 tablet_bootstrap.cc:654] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:13.127988 29529 log.cc:824] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:13.128592 29531 tablet_bootstrap.cc:492] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920: Bootstrap starting.
I20260805 23:09:13.128877 29530 tablet_bootstrap.cc:654] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:13.129552 29530 log.cc:824] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:13.130254 29529 tablet_bootstrap.cc:492] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: No bootstrap required, opened a new log
I20260805 23:09:13.130447 29529 ts_tablet_manager.cc:1397] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:09:13.130784 29531 tablet_bootstrap.cc:654] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:13.131429 29530 tablet_bootstrap.cc:492] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: No bootstrap required, opened a new log
I20260805 23:09:13.131620 29530 ts_tablet_manager.cc:1397] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260805 23:09:13.131646 29531 log.cc:824] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:13.134110 29529 raft_consensus.cc:348] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.134320 29529 raft_consensus.cc:374] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:13.134352 29529 raft_consensus.cc:729] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31854ac9e71c417cb4848ad6d188cbce, State: Initialized, Role: FOLLOWER
I20260805 23:09:13.134827 29529 consensus_queue.cc:260] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.134979 29530 raft_consensus.cc:348] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.135207 29530 raft_consensus.cc:374] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:13.135267 29530 raft_consensus.cc:729] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e8db3cc1f34b4d9fd2196e0b7c9a3d, State: Initialized, Role: FOLLOWER
I20260805 23:09:13.135512 29504 heartbeater.cc:503] Master 127.3.12.126:42957 was elected leader, sending a full tablet report...
I20260805 23:09:13.135711 29530 consensus_queue.cc:260] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.135833 29529 ts_tablet_manager.cc:1428] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:09:13.136373 29320 heartbeater.cc:503] Master 127.3.12.126:42957 was elected leader, sending a full tablet report...
I20260805 23:09:13.136745 29530 ts_tablet_manager.cc:1428] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260805 23:09:13.137984 29531 tablet_bootstrap.cc:492] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920: No bootstrap required, opened a new log
I20260805 23:09:13.138230 29531 ts_tablet_manager.cc:1397] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260805 23:09:13.142056 29531 raft_consensus.cc:348] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.142344 29531 raft_consensus.cc:374] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:13.142443 29531 raft_consensus.cc:729] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30002f680f8f4632838b2d94de0c0920, State: Initialized, Role: FOLLOWER
I20260805 23:09:13.142881 29531 consensus_queue.cc:260] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.142946 29536 raft_consensus.cc:482] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:09:13.143107 29536 raft_consensus.cc:504] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.143647 29133 heartbeater.cc:503] Master 127.3.12.126:42957 was elected leader, sending a full tablet report...
I20260805 23:09:13.143913 29531 ts_tablet_manager.cc:1428] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 23:09:13.144457 29536 leader_election.cc:288] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30002f680f8f4632838b2d94de0c0920 (127.3.12.65:41405), 31854ac9e71c417cb4848ad6d188cbce (127.3.12.67:44725)
W20260805 23:09:13.149137 29137 tablet.cc:2369] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:09:13.149591 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.144641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:41405 (local address 127.3.12.66:56911)
0805 23:09:13.144790 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:13.144807 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:13.144881 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:13.145323 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:13.145327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:13.145347 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:13.145500 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.145508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.146546 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.146551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.147881 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.147893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.148932 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.148936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.149151 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.149273 (+ 122us) client_negotiation.cc:769] Sending connection context
0805 23:09:13.149362 (+ 89us) client_negotiation.cc:240] Negotiation successful
0805 23:09:13.149452 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":154,"spinlock_wait_cycles":34048}
I20260805 23:09:13.149590 29523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.144687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:56911 (local address 127.3.12.65:41405)
0805 23:09:13.144820 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:13.144824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:13.144856 (+ 32us) server_negotiation.cc:407] Connection header received
0805 23:09:13.145025 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:13.145031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:13.145121 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:13.145194 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:13.145702 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.146424 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.148160 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.148795 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.148936 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.148986 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:13.149389 (+ 403us) server_negotiation.cc:299] Negotiation successful
0805 23:09:13.149468 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 23:09:13.149591 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.145300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:52663 (local address 127.3.12.67:44725)
0805 23:09:13.145410 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:13.145414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:13.145427 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:09:13.145509 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:13.145513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:13.145575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:13.145647 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:13.145910 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.146509 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.148032 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.148800 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.148920 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.148982 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:13.149386 (+ 404us) server_negotiation.cc:299] Negotiation successful
0805 23:09:13.149468 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260805 23:09:13.150244 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.144904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:44725 (local address 127.3.12.66:52663)
0805 23:09:13.145269 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:13.145284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:13.145411 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:13.145666 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:13.145668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:13.145683 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:13.145799 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.145804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.146660 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.146662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.147885 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:13.147893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.148933 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.148935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:13.149145 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.149224 (+ 79us) client_negotiation.cc:769] Sending connection context
0805 23:09:13.149985 (+ 761us) client_negotiation.cc:240] Negotiation successful
0805 23:09:13.150095 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":91,"thread_start_us":122,"threads_started":1}
I20260805 23:09:13.150364 29064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc42c549c0274da0b339cb547d8aa6b4" candidate_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30002f680f8f4632838b2d94de0c0920" is_pre_election: true
I20260805 23:09:13.150682 29064 raft_consensus.cc:2454] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41e8db3cc1f34b4d9fd2196e0b7c9a3d in term 0.
I20260805 23:09:13.151011 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc42c549c0274da0b339cb547d8aa6b4" candidate_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31854ac9e71c417cb4848ad6d188cbce" is_pre_election: true
I20260805 23:09:13.151208 29168 leader_election.cc:302] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30002f680f8f4632838b2d94de0c0920, 41e8db3cc1f34b4d9fd2196e0b7c9a3d; no voters:
I20260805 23:09:13.151296 29430 raft_consensus.cc:2454] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41e8db3cc1f34b4d9fd2196e0b7c9a3d in term 0.
I20260805 23:09:13.151516 29536 raft_consensus.cc:2781] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:09:13.151618 29536 raft_consensus.cc:482] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:09:13.151690 29536 raft_consensus.cc:3037] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:13.152868 29536 raft_consensus.cc:504] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.153198 29536 leader_election.cc:288] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [CANDIDATE]: Term 1 election: Requested vote from peers 30002f680f8f4632838b2d94de0c0920 (127.3.12.65:41405), 31854ac9e71c417cb4848ad6d188cbce (127.3.12.67:44725)
I20260805 23:09:13.153550 29064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc42c549c0274da0b339cb547d8aa6b4" candidate_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30002f680f8f4632838b2d94de0c0920"
I20260805 23:09:13.153575 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc42c549c0274da0b339cb547d8aa6b4" candidate_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31854ac9e71c417cb4848ad6d188cbce"
I20260805 23:09:13.153677 29064 raft_consensus.cc:3037] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:13.153677 29430 raft_consensus.cc:3037] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:13.154928 29430 raft_consensus.cc:2454] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41e8db3cc1f34b4d9fd2196e0b7c9a3d in term 1.
I20260805 23:09:13.154928 29064 raft_consensus.cc:2454] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41e8db3cc1f34b4d9fd2196e0b7c9a3d in term 1.
I20260805 23:09:13.155233 29168 leader_election.cc:302] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30002f680f8f4632838b2d94de0c0920, 41e8db3cc1f34b4d9fd2196e0b7c9a3d; no voters:
I20260805 23:09:13.155413 29536 raft_consensus.cc:2781] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:13.155646 29536 raft_consensus.cc:686] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 1 LEADER]: Becoming Leader. State: Replica: 41e8db3cc1f34b4d9fd2196e0b7c9a3d, State: Running, Role: LEADER
I20260805 23:09:13.155994 29536 consensus_queue.cc:237] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } }
I20260805 23:09:13.159096 28875 catalog_manager.cc:5495] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 41e8db3cc1f34b4d9fd2196e0b7c9a3d (127.3.12.66). New cstate: current_term: 1 leader_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41e8db3cc1f34b4d9fd2196e0b7c9a3d" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38677 } health_report { overall_health: HEALTHY } } }
I20260805 23:09:13.179517 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.175731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.3.12.66:38677)
0805 23:09:13.175889 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:13.175896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:13.175914 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:09:13.176020 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:13.176023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:13.176082 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:13.176212 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:13.176488 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.177018 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.177906 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:13.178168 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:13.178286 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:13.178625 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:13.179103 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:13.179246 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:13.179308 (+ 62us) server_negotiation.cc:299] Negotiation successful
0805 23:09:13.179364 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260805 23:09:13.185910 29064 raft_consensus.cc:1261] T cc42c549c0274da0b339cb547d8aa6b4 P 30002f680f8f4632838b2d94de0c0920 [term 1 FOLLOWER]: Refusing update from remote peer 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:13.185956 29430 raft_consensus.cc:1261] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 1 FOLLOWER]: Refusing update from remote peer 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:13.186712 29536 consensus_queue.cc:1035] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31854ac9e71c417cb4848ad6d188cbce" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:13.186949 29540 consensus_queue.cc:1035] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "30002f680f8f4632838b2d94de0c0920" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:13.194823 29544 mvcc.cc:205] Tried to move back new op lower bound from 7315338662641221632 to 7315338662529982464. Current Snapshot: MvccSnapshot[applied={T|T < 7315338662641221632}]
I20260805 23:09:13.194887 29545 mvcc.cc:205] Tried to move back new op lower bound from 7315338662641221632 to 7315338662529982464. Current Snapshot: MvccSnapshot[applied={T|T < 7315338662641221632}]
I20260805 23:09:13.204571 28875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:13.204754 28875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:13.207322 28875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 909e998ede6b43518fd962a5cc493953: Sending DeleteTablet for 3 replicas of tablet cc42c549c0274da0b339cb547d8aa6b4
I20260805 23:09:13.208174 29401 tablet_service.cc:1514] Processing DeleteTablet for tablet cc42c549c0274da0b339cb547d8aa6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:13 UTC) from {username='slave'} at 127.0.0.1:41462
I20260805 23:09:13.208189 29231 tablet_service.cc:1514] Processing DeleteTablet for tablet cc42c549c0274da0b339cb547d8aa6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:13 UTC) from {username='slave'} at 127.0.0.1:57374
I20260805 23:09:13.208741 29558 tablet_replica.cc:330] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: stopping tablet replica
I20260805 23:09:13.208957 29038 tablet_service.cc:1514] Processing DeleteTablet for tablet cc42c549c0274da0b339cb547d8aa6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:13 UTC) from {username='slave'} at 127.0.0.1:49186
I20260805 23:09:13.209234 29558 raft_consensus.cc:2229] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:09:13.209501 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 28957
I20260805 23:09:13.209678 29558 raft_consensus.cc:2258] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:13.212576 29555 tablet_replica.cc:330] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: stopping tablet replica
I20260805 23:09:13.212883 29555 raft_consensus.cc:2229] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:13.213164 29555 raft_consensus.cc:2258] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:13.215574 29555 ts_tablet_manager.cc:1905] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:13.218806 29555 ts_tablet_manager.cc:1918] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:13.218911 29555 log.cc:1196] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/cc42c549c0274da0b339cb547d8aa6b4
I20260805 23:09:13.219269 29555 ts_tablet_manager.cc:1939] T cc42c549c0274da0b339cb547d8aa6b4 P 31854ac9e71c417cb4848ad6d188cbce: Deleting consensus metadata
I20260805 23:09:13.219808 29558 ts_tablet_manager.cc:1905] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:13.221315 28862 catalog_manager.cc:4841] TS 31854ac9e71c417cb4848ad6d188cbce (127.3.12.67:44725): tablet cc42c549c0274da0b339cb547d8aa6b4 (table test-table [id=398cd21764c745ef967c6a08aacb9a8f]) successfully deleted
I20260805 23:09:13.222622 29558 ts_tablet_manager.cc:1918] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:13.222747 29558 log.cc:1196] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/cc42c549c0274da0b339cb547d8aa6b4
I20260805 23:09:13.223042 29558 ts_tablet_manager.cc:1939] T cc42c549c0274da0b339cb547d8aa6b4 P 41e8db3cc1f34b4d9fd2196e0b7c9a3d: Deleting consensus metadata
I20260805 23:09:13.224129 28861 catalog_manager.cc:4841] TS 41e8db3cc1f34b4d9fd2196e0b7c9a3d (127.3.12.66:38677): tablet cc42c549c0274da0b339cb547d8aa6b4 (table test-table [id=398cd21764c745ef967c6a08aacb9a8f]) successfully deleted
W20260805 23:09:13.225540 28862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:41405 (error 108)
I20260805 23:09:13.226506 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:13.226246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:41405 (local address 127.0.0.1:49198)
0805 23:09:13.226374 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:13.226435 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:41405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260805 23:09:13.226706 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 29139
W20260805 23:09:13.226707 28862 catalog_manager.cc:4584] TS 30002f680f8f4632838b2d94de0c0920 (127.3.12.65:41405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc42c549c0274da0b339cb547d8aa6b4: Network error: Client connection negotiation failed: client connection to 127.3.12.65:41405: connect: Connection refused (error 111)
I20260805 23:09:13.236569 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 29323
I20260805 23:09:13.246714 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 28837
2026-08-05T23:09:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:13 dist-test-slave-2d19 krb5kdc[29565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:13 dist-test-slave-2d19 krb5kdc[29565](info): set up 2 sockets
Aug 05 23:09:13 dist-test-slave-2d19 krb5kdc[29565](info): commencing operation
krb5kdc: starting...
W20260805 23:09:14.494369 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:14 dist-test-slave-2d19 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:14.629460 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36201
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:36201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:14.742367 29592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:14.742638 29592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:14.742679 29592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:14.746651 29592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:14.746711 29592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:14.746730 29592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:14.746747 29592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:14.746762 29592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:14.752002 29592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:36201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:14.753141 29592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:14.754130 29592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:09:14.763062 29592 server_base.cc:1034] running on GCE node
W20260805 23:09:14.762975 29600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:14.762982 29601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:14.762975 29603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:14.763886 29592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:14.765053 29592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:14.766244 29592 hybrid_clock.cc:648] HybridClock initialized: now 1785971354766219 us; error 40 us; skew 500 ppm
Aug 05 23:09:14 dist-test-slave-2d19 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971354, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:14.769038 29592 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:14.769965 29592 webserver.cc:460] Webserver started at http://127.3.12.126:45505/ using document root <none> and password file <none>
I20260805 23:09:14.770532 29592 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:14.770645 29592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:14.770886 29592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:14.773059 29592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "31387412d8bb434e802b818d62194440"
format_stamp: "Formatted at 2026-08-05 23:09:14 on dist-test-slave-2d19"
server_key: "b318a4db6729c6787f714e421a797358"
server_key_iv: "2f44587fc7f0c980d23e3abf5e189c90"
server_key_version: "encryptionkey@0"
I20260805 23:09:14.773633 29592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "31387412d8bb434e802b818d62194440"
format_stamp: "Formatted at 2026-08-05 23:09:14 on dist-test-slave-2d19"
server_key: "b318a4db6729c6787f714e421a797358"
server_key_iv: "2f44587fc7f0c980d23e3abf5e189c90"
server_key_version: "encryptionkey@0"
I20260805 23:09:14.777417 29592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260805 23:09:14.779870 29615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:14.781168 29592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:14.781315 29592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "31387412d8bb434e802b818d62194440"
format_stamp: "Formatted at 2026-08-05 23:09:14 on dist-test-slave-2d19"
server_key: "b318a4db6729c6787f714e421a797358"
server_key_iv: "2f44587fc7f0c980d23e3abf5e189c90"
server_key_version: "encryptionkey@0"
I20260805 23:09:14.781431 29592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:14.807106 29592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:14.810247 29592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:14.810464 29592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:14.818603 29592 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:36201
I20260805 23:09:14.818634 29690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:36201 every 8 connection(s)
I20260805 23:09:14.819602 29592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:14.822515 29691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:14.827337 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 29592
I20260805 23:09:14.827489 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:14.827811 3121 external_mini_cluster.cc:1428] Setting key 99328ef14d03ec52555b646830535972
I20260805 23:09:14.828557 29691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440: Bootstrap starting.
I20260805 23:09:14.831039 29691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:14.831813 29691 log.cc:824] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:14.834087 29691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440: No bootstrap required, opened a new log
Aug 05 23:09:14 dist-test-slave-2d19 krb5kdc[29565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:14.838932 29691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31387412d8bb434e802b818d62194440" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36201 } }
I20260805 23:09:14.839234 29691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:14.839308 29691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31387412d8bb434e802b818d62194440, State: Initialized, Role: FOLLOWER
I20260805 23:09:14.839794 29691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31387412d8bb434e802b818d62194440" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36201 } }
I20260805 23:09:14.839968 29691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:14.840047 29691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:14.840159 29691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:14.841511 29691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31387412d8bb434e802b818d62194440" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36201 } }
I20260805 23:09:14.841871 29691 leader_election.cc:302] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31387412d8bb434e802b818d62194440; no voters:
I20260805 23:09:14.842243 29691 leader_election.cc:288] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:14.842370 29696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:14.842613 29696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [term 1 LEADER]: Becoming Leader. State: Replica: 31387412d8bb434e802b818d62194440, State: Running, Role: LEADER
I20260805 23:09:14.842988 29696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31387412d8bb434e802b818d62194440" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36201 } }
I20260805 23:09:14.843624 29691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:14.843856 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:14.829342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.3.12.126:36201)
0805 23:09:14.829806 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:14.829822 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:14.829858 (+ 36us) server_negotiation.cc:407] Connection header received
0805 23:09:14.830487 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:14.830504 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:14.830852 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:14.831210 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:14.832078 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:14.833469 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:14.834396 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:14.835098 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:14.835265 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:14.838245 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:14.838251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:14.838266 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:14.838294 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:14.840306 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:14.840893 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:14.840899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:14.840905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:14.841050 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:14.841372 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:14.841377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:14.841378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:14.841874 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:14.842117 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:14.842423 (+ 306us) server_negotiation.cc:299] Negotiation successful
0805 23:09:14.842903 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":141,"threads_started":1}
I20260805 23:09:14.844216 29696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31387412d8bb434e802b818d62194440. Latest consensus state: current_term: 1 leader_uuid: "31387412d8bb434e802b818d62194440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31387412d8bb434e802b818d62194440" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36201 } } }
I20260805 23:09:14.844370 29696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:14.844908 29699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31387412d8bb434e802b818d62194440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31387412d8bb434e802b818d62194440" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36201 } } }
I20260805 23:09:14.845085 29699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:14.846325 29704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:14.849397 29704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:14.855216 29704 catalog_manager.cc:1347] Generated new cluster ID: c7fcd03853c044a7bd13486bf6538b65
I20260805 23:09:14.855293 29704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:14.866896 29704 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:14.868216 29704 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:14.878077 29704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440: Generated new TSK 0
I20260805 23:09:14.878942 29704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:09:14.919293 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36201
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:15.035260 29728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:15.035567 29728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:15.035655 29728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:15.039613 29728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:15.039670 29728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:15.039727 29728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:09:15.045153 29728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:15.046262 29728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:15.047241 29728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:15.056916 29740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:15.056936 29736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:15.057206 29728 server_base.cc:1034] running on GCE node
W20260805 23:09:15.056890 29734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:15.058260 29728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:15.059058 29728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:15.060227 29728 hybrid_clock.cc:648] HybridClock initialized: now 1785971355060212 us; error 34 us; skew 500 ppm
Aug 05 23:09:15 dist-test-slave-2d19 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:15.063207 29728 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:09:15.064117 29728 webserver.cc:460] Webserver started at http://127.3.12.65:41349/ using document root <none> and password file <none>
I20260805 23:09:15.064759 29728 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:15.064857 29728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:15.065097 29728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:15.067315 29728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "ade2c40bf839208ce6d2738ecbaaf29d"
server_key_iv: "2e84550838971a722580785f434ca687"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.067905 29728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "ade2c40bf839208ce6d2738ecbaaf29d"
server_key_iv: "2e84550838971a722580785f434ca687"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.071765 29728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:09:15.074296 29754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.075495 29728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:15.075637 29728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "ade2c40bf839208ce6d2738ecbaaf29d"
server_key_iv: "2e84550838971a722580785f434ca687"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.075754 29728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:15.094439 29728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:15.097617 29728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:15.097833 29728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:15.098448 29728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:15.099328 29728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:15.099404 29728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.099488 29728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:15.099539 29728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.108819 29728 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:46595
I20260805 23:09:15.108843 29905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:46595 every 8 connection(s)
I20260805 23:09:15.109771 29728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:09:15.117130 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 29728
I20260805 23:09:15.117265 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:09:15.117578 3121 external_mini_cluster.cc:1428] Setting key 87c8ee21d2130aa6ccf859a4e180d8b7
Aug 05 23:09:15 dist-test-slave-2d19 krb5kdc[29565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:15.123682 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.111682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:48293 (local address 127.3.12.126:36201)
0805 23:09:15.111836 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.111839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.112777 (+ 938us) server_negotiation.cc:407] Connection header received
0805 23:09:15.114060 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.114064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.114121 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.114207 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:15.115372 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.115915 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.117105 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.117453 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.117595 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.120043 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:15.120047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:15.120050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:15.120076 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:15.121592 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:15.122110 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:15.122115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:15.122117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:15.122203 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:15.122547 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:15.122559 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:15.122561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:15.122745 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:15.122856 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.123352 (+ 496us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.123494 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260805 23:09:15.124539 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.112097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:36201 (local address 127.3.12.65:48293)
0805 23:09:15.112613 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.112656 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:15.113841 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:15.114377 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:15.114384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:15.114739 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:15.115211 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.115224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.116062 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.116066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.116963 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.116972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.117639 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.117643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.117815 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.118437 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:15.118453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:15.119880 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:15.121712 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:15.121719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:15.121735 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:15.121997 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:15.122298 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:15.122303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:15.122305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:15.122449 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:15.122853 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:15.122860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:15.123120 (+ 260us) client_negotiation.cc:769] Sending connection context
0805 23:09:15.123311 (+ 191us) client_negotiation.cc:240] Negotiation successful
0805 23:09:15.123633 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":180,"threads_started":1}
I20260805 23:09:15.125835 29906 heartbeater.cc:347] Connected to a master server at 127.3.12.126:36201
I20260805 23:09:15.126180 29906 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:15.126878 29906 heartbeater.cc:511] Master 127.3.12.126:36201 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:09:15.128670 29643 ts_manager.cc:194] Registered new tserver with Master: e0bc8b0fce694e2fa6d6ca4d8d60ad25 (127.3.12.65:46595)
I20260805 23:09:15.129825 29643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:48293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:09:15.147699 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36201
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:15.264446 29914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:15.264703 29914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:15.264782 29914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:15.268970 29914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:15.269026 29914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:15.269131 29914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:09:15.274513 29914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:15.275588 29914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:15.276721 29914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:15.286475 29922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:15.286484 29925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:15.286514 29921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:15.286648 29914 server_base.cc:1034] running on GCE node
I20260805 23:09:15.287194 29914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:15.287839 29914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:15.289004 29914 hybrid_clock.cc:648] HybridClock initialized: now 1785971355288972 us; error 44 us; skew 500 ppm
Aug 05 23:09:15 dist-test-slave-2d19 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:15.291855 29914 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:09:15.292851 29914 webserver.cc:460] Webserver started at http://127.3.12.66:38147/ using document root <none> and password file <none>
I20260805 23:09:15.293455 29914 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:15.293552 29914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:15.293844 29914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:15.296017 29914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "972846a3e3fc4d5fa878fca8bf038b18"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "7c4df8c3ee0755211e951e121d4cb83e"
server_key_iv: "dbbb33c4a9cb585dfa6bcdd087a5e58c"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.296634 29914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "972846a3e3fc4d5fa878fca8bf038b18"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "7c4df8c3ee0755211e951e121d4cb83e"
server_key_iv: "dbbb33c4a9cb585dfa6bcdd087a5e58c"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.300215 29914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:09:15.302692 29932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.303900 29914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:15.304049 29914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "972846a3e3fc4d5fa878fca8bf038b18"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "7c4df8c3ee0755211e951e121d4cb83e"
server_key_iv: "dbbb33c4a9cb585dfa6bcdd087a5e58c"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.304162 29914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:15.331220 29914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:15.334710 29914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:15.334925 29914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:15.335604 29914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:15.336527 29914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:15.336599 29914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.336676 29914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:15.336720 29914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.346539 29914 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:38215
I20260805 23:09:15.346558 30078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:38215 every 8 connection(s)
I20260805 23:09:15.347527 29914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:09:15.355247 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 29914
I20260805 23:09:15.355373 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:09:15.355710 3121 external_mini_cluster.cc:1428] Setting key 5667d2e9c42d7f0b34bf343837669214
Aug 05 23:09:15 dist-test-slave-2d19 krb5kdc[29565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:15.361718 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.349341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:59855 (local address 127.3.12.126:36201)
0805 23:09:15.349488 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.349491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.350570 (+ 1079us) server_negotiation.cc:407] Connection header received
0805 23:09:15.351831 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.351834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.351880 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.351968 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:15.353237 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.353750 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.355095 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.355431 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.355544 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.358135 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:15.358144 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:15.358147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:15.358171 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:15.359552 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:15.360073 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:15.360080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:15.360082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:15.360157 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:15.360546 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:15.360551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:15.360552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:15.360738 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:15.360844 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.361490 (+ 646us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.361598 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:09:15.362964 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.349763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:36201 (local address 127.3.12.66:59855)
0805 23:09:15.350406 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.350454 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:15.351642 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:15.352155 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:15.352161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:15.352621 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:15.353105 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.353119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.353945 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.353952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.354890 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.354948 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.355594 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.355597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.355778 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.356493 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:15.356509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:15.357966 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:15.359674 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:15.359680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:15.359697 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:15.359958 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:15.360284 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:15.360288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:15.360290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:15.360442 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:15.360855 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:15.360865 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:15.361189 (+ 324us) client_negotiation.cc:769] Sending connection context
0805 23:09:15.361394 (+ 205us) client_negotiation.cc:240] Negotiation successful
0805 23:09:15.361772 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":119,"threads_started":1}
I20260805 23:09:15.364353 30079 heartbeater.cc:347] Connected to a master server at 127.3.12.126:36201
I20260805 23:09:15.364627 30079 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:15.365144 30079 heartbeater.cc:511] Master 127.3.12.126:36201 requested a full tablet report, sending...
I20260805 23:09:15.366256 29643 ts_manager.cc:194] Registered new tserver with Master: 972846a3e3fc4d5fa878fca8bf038b18 (127.3.12.66:38215)
I20260805 23:09:15.366833 29643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:59855
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:09:15.387144 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36201
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:15.504379 30085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:15.504700 30085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:15.504791 30085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:15.508991 30085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:15.509079 30085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:15.509172 30085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:09:15.514710 30085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:36201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:15.515846 30085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:15.516969 30085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:15.527036 30095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:15.527099 30092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:15.527027 30093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:15.527350 30085 server_base.cc:1034] running on GCE node
I20260805 23:09:15.527947 30085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:15.528705 30085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:15.529915 30085 hybrid_clock.cc:648] HybridClock initialized: now 1785971355529892 us; error 33 us; skew 500 ppm
Aug 05 23:09:15 dist-test-slave-2d19 krb5kdc[29565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:15.532922 30085 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:09:15.534011 30085 webserver.cc:460] Webserver started at http://127.3.12.67:38841/ using document root <none> and password file <none>
I20260805 23:09:15.534818 30085 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:15.534909 30085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:15.535161 30085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:15.537351 30085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "390fa68b41b44552a8d258dd75625f22"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "4c9f08e61ab356a7716d2ed3ab55ddc3"
server_key_iv: "da6229892628d4cf48bf1374f1209592"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.537940 30085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "390fa68b41b44552a8d258dd75625f22"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "4c9f08e61ab356a7716d2ed3ab55ddc3"
server_key_iv: "da6229892628d4cf48bf1374f1209592"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.542373 30085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:15.545320 30105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.546658 30085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260805 23:09:15.546814 30085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "390fa68b41b44552a8d258dd75625f22"
format_stamp: "Formatted at 2026-08-05 23:09:15 on dist-test-slave-2d19"
server_key: "4c9f08e61ab356a7716d2ed3ab55ddc3"
server_key_iv: "da6229892628d4cf48bf1374f1209592"
server_key_version: "encryptionkey@0"
I20260805 23:09:15.546944 30085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:15.574460 30085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:15.577996 30085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:15.578231 30085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:15.578928 30085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:15.579909 30085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:15.579982 30085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.580063 30085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:15.580117 30085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:15.589995 30085 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:33121
I20260805 23:09:15.590019 30263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:33121 every 8 connection(s)
I20260805 23:09:15.590955 30085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:09:15.594668 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 30085
I20260805 23:09:15.594779 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:09:15.595098 3121 external_mini_cluster.cc:1428] Setting key 66b522cc30997c8d5b4704f9817ff7e9
Aug 05 23:09:15 dist-test-slave-2d19 krb5kdc[29565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:15.604997 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.592864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:33197 (local address 127.3.12.126:36201)
0805 23:09:15.593008 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.593012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.593867 (+ 855us) server_negotiation.cc:407] Connection header received
0805 23:09:15.595344 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.595348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.595412 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.595499 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:15.596972 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.597602 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.598878 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.599085 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.599160 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.601651 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:15.601654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:15.601656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:15.601680 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:15.602992 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:15.603485 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:15.603487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:15.603489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:15.603575 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:15.603950 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:15.603952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:15.603954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:15.604132 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:15.604211 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.604771 (+ 560us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.604875 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:09:15.606323 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.593248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:36201 (local address 127.3.12.67:33197)
0805 23:09:15.593709 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.593753 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:15.595088 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:15.595679 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:15.595686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:15.596155 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:15.596809 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.596830 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.597732 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.597735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.598770 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.598780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.599216 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.599219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.599361 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.600005 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:15.600030 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:15.601490 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:15.603112 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:15.603117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:15.603129 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:15.603394 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:15.603686 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:15.603689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:15.603690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:15.603856 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:15.604278 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:15.604284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:15.604547 (+ 263us) client_negotiation.cc:769] Sending connection context
0805 23:09:15.604719 (+ 172us) client_negotiation.cc:240] Negotiation successful
0805 23:09:15.605023 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260805 23:09:15.607612 30264 heartbeater.cc:347] Connected to a master server at 127.3.12.126:36201
I20260805 23:09:15.607870 30264 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:15.608517 30264 heartbeater.cc:511] Master 127.3.12.126:36201 requested a full tablet report, sending...
I20260805 23:09:15.609653 29643 ts_manager.cc:194] Registered new tserver with Master: 390fa68b41b44552a8d258dd75625f22 (127.3.12.67:33121)
I20260805 23:09:15.610208 29643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:33197
I20260805 23:09:15.621220 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:09:15.631353 29694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.623238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.3.12.126:36201)
0805 23:09:15.623367 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.623370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.623536 (+ 166us) server_negotiation.cc:407] Connection header received
0805 23:09:15.623647 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.623649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.623688 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.623755 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:15.624527 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.625036 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.625965 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.626174 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.626256 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.627780 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:15.627783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:15.627785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:15.627809 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:15.629436 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:15.629999 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:15.630002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:15.630003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:15.630088 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:15.630494 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:15.630497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:15.630498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:15.630704 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:15.630828 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.631148 (+ 320us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.631233 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:09:15.634619 29643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:09:15.637749 29643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:09:15.651968 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.646036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38215 (local address 127.0.0.1:52386)
0805 23:09:15.646374 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.646421 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:15.646547 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:15.647464 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:15.647472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:15.647495 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:15.647654 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.647663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.648951 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.648956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.649937 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.649949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.651416 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.651420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.651515 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.651544 (+ 29us) client_negotiation.cc:769] Sending connection context
0805 23:09:15.651660 (+ 116us) client_negotiation.cc:240] Negotiation successful
0805 23:09:15.651801 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260805 23:09:15.652082 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.646201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.3.12.66:38215)
0805 23:09:15.646742 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.646756 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.646781 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:09:15.646864 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.646869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.647087 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.647255 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:15.647791 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.648820 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.650133 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.651019 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.651152 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.651350 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.651710 (+ 360us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.651924 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":124,"threads_started":1}
I20260805 23:09:15.652931 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.646779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:46595 (local address 127.0.0.1:39312)
0805 23:09:15.647288 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.647303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:15.647392 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:15.648301 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:15.648304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:15.648322 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:15.648401 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.648406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.649768 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.649772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.650992 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.651002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.652304 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.652308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.652572 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.652636 (+ 64us) client_negotiation.cc:769] Sending connection context
0805 23:09:15.652713 (+ 77us) client_negotiation.cc:240] Negotiation successful
0805 23:09:15.652800 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":119,"thread_start_us":97,"threads_started":1}
I20260805 23:09:15.653087 30283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.646870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.3.12.67:33121)
0805 23:09:15.647643 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.647655 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.647677 (+ 22us) server_negotiation.cc:407] Connection header received
0805 23:09:15.647755 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.647760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.647942 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.648102 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:15.648373 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.649593 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.651183 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.651974 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.652069 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.652208 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.652797 (+ 589us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.652895 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":139,"threads_started":1}
I20260805 23:09:15.653412 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.646277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:33121 (local address 127.0.0.1:55410)
0805 23:09:15.647215 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.647230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:15.647323 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:15.648068 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:15.648072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:15.648090 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:15.648206 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.648213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.649747 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.649751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.651081 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.651094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.652304 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.652308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.652567 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.652639 (+ 72us) client_negotiation.cc:769] Sending connection context
0805 23:09:15.652713 (+ 74us) client_negotiation.cc:240] Negotiation successful
0805 23:09:15.653301 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":665,"thread_start_us":91,"threads_started":1}
I20260805 23:09:15.653679 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.646956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.3.12.65:46595)
0805 23:09:15.647713 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.647720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.647745 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:09:15.647824 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.647830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.648049 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.648195 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:15.648523 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.649640 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.651118 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.651953 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.652069 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.652208 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.653120 (+ 912us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.653223 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":125,"threads_started":1}
I20260805 23:09:15.655735 29989 tablet_service.cc:1467] Processing CreateTablet for tablet 415eac4e6b3b440b81a07e5f941ccaf0 (DEFAULT_TABLE table=test-table [id=868b09c0dee54f908530f351c552a242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:15.656090 30168 tablet_service.cc:1467] Processing CreateTablet for tablet 415eac4e6b3b440b81a07e5f941ccaf0 (DEFAULT_TABLE table=test-table [id=868b09c0dee54f908530f351c552a242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:15.656987 29989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415eac4e6b3b440b81a07e5f941ccaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:15.657059 30168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415eac4e6b3b440b81a07e5f941ccaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:15.659150 29816 tablet_service.cc:1467] Processing CreateTablet for tablet 415eac4e6b3b440b81a07e5f941ccaf0 (DEFAULT_TABLE table=test-table [id=868b09c0dee54f908530f351c552a242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:15.660048 29816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415eac4e6b3b440b81a07e5f941ccaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:15.664608 30288 tablet_bootstrap.cc:492] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18: Bootstrap starting.
I20260805 23:09:15.665113 30289 tablet_bootstrap.cc:492] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: Bootstrap starting.
I20260805 23:09:15.666826 30290 tablet_bootstrap.cc:492] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25: Bootstrap starting.
I20260805 23:09:15.667824 30289 tablet_bootstrap.cc:654] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:15.668661 30289 log.cc:824] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:15.668715 30290 tablet_bootstrap.cc:654] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:15.669401 30290 log.cc:824] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:15.670837 30289 tablet_bootstrap.cc:492] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: No bootstrap required, opened a new log
I20260805 23:09:15.671036 30289 ts_tablet_manager.cc:1397] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260805 23:09:15.671437 30288 tablet_bootstrap.cc:654] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:15.671447 30290 tablet_bootstrap.cc:492] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25: No bootstrap required, opened a new log
I20260805 23:09:15.671824 30290 ts_tablet_manager.cc:1397] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:09:15.672322 30288 log.cc:824] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:15.673990 30288 tablet_bootstrap.cc:492] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18: No bootstrap required, opened a new log
I20260805 23:09:15.674203 30288 ts_tablet_manager.cc:1397] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260805 23:09:15.675472 30289 raft_consensus.cc:348] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.675786 30289 raft_consensus.cc:374] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:15.675844 30289 raft_consensus.cc:729] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390fa68b41b44552a8d258dd75625f22, State: Initialized, Role: FOLLOWER
I20260805 23:09:15.676355 30289 consensus_queue.cc:260] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.676445 30290 raft_consensus.cc:348] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.676704 30290 raft_consensus.cc:374] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:15.676769 30290 raft_consensus.cc:729] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0bc8b0fce694e2fa6d6ca4d8d60ad25, State: Initialized, Role: FOLLOWER
I20260805 23:09:15.677188 30290 consensus_queue.cc:260] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.677294 30264 heartbeater.cc:503] Master 127.3.12.126:36201 was elected leader, sending a full tablet report...
I20260805 23:09:15.677459 30289 ts_tablet_manager.cc:1428] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 23:09:15.677534 30288 raft_consensus.cc:348] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.677776 30288 raft_consensus.cc:374] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:15.677831 30288 raft_consensus.cc:729] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972846a3e3fc4d5fa878fca8bf038b18, State: Initialized, Role: FOLLOWER
I20260805 23:09:15.678047 30290 ts_tablet_manager.cc:1428] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 23:09:15.678228 30288 consensus_queue.cc:260] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.678382 29906 heartbeater.cc:503] Master 127.3.12.126:36201 was elected leader, sending a full tablet report...
I20260805 23:09:15.679136 30288 ts_tablet_manager.cc:1428] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:09:15.679481 30079 heartbeater.cc:503] Master 127.3.12.126:36201 was elected leader, sending a full tablet report...
I20260805 23:09:15.737756 30294 raft_consensus.cc:482] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:09:15.737962 30294 raft_consensus.cc:504] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.739132 30294 leader_election.cc:288] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 972846a3e3fc4d5fa878fca8bf038b18 (127.3.12.66:38215), e0bc8b0fce694e2fa6d6ca4d8d60ad25 (127.3.12.65:46595)
I20260805 23:09:15.743315 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.739390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38215 (local address 127.3.12.67:44667)
0805 23:09:15.739527 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.739545 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:15.739650 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:15.740034 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:15.740038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:15.740059 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:15.740178 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.740184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.741172 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.741176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.742079 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.742089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.742849 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.742852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.742963 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.742990 (+ 27us) client_negotiation.cc:769] Sending connection context
0805 23:09:15.743077 (+ 87us) client_negotiation.cc:240] Negotiation successful
0805 23:09:15.743160 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260805 23:09:15.743322 30279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.739519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:44667 (local address 127.3.12.66:38215)
0805 23:09:15.739681 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.739687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.739716 (+ 29us) server_negotiation.cc:407] Connection header received
0805 23:09:15.739787 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.739793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.739847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.739959 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:15.740330 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.741056 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.742206 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.742725 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.742807 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.742860 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.743097 (+ 237us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.743186 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:09:15.743615 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.739708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:46595 (local address 127.3.12.67:40899)
0805 23:09:15.740159 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.740171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:15.740316 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:15.740548 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:15.740550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:15.740570 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:15.740685 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.740691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.741512 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.741515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.742386 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:15.742396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.743177 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.743180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:15.743283 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.743305 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:09:15.743388 (+ 83us) client_negotiation.cc:240] Negotiation successful
0805 23:09:15.743458 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":149,"threads_started":1}
I20260805 23:09:15.743969 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.739880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:40899 (local address 127.3.12.65:46595)
0805 23:09:15.740045 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.740060 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.740284 (+ 224us) server_negotiation.cc:407] Connection header received
0805 23:09:15.740405 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.740407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.740448 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.740506 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:15.740793 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.741390 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.742513 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.743031 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.743151 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.743206 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.743785 (+ 579us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.743872 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260805 23:09:15.744156 30011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415eac4e6b3b440b81a07e5f941ccaf0" candidate_uuid: "390fa68b41b44552a8d258dd75625f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972846a3e3fc4d5fa878fca8bf038b18" is_pre_election: true
I20260805 23:09:15.744537 30011 raft_consensus.cc:2454] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390fa68b41b44552a8d258dd75625f22 in term 0.
I20260805 23:09:15.744541 29843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415eac4e6b3b440b81a07e5f941ccaf0" candidate_uuid: "390fa68b41b44552a8d258dd75625f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" is_pre_election: true
I20260805 23:09:15.744818 29843 raft_consensus.cc:2454] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390fa68b41b44552a8d258dd75625f22 in term 0.
I20260805 23:09:15.745083 30109 leader_election.cc:302] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390fa68b41b44552a8d258dd75625f22, 972846a3e3fc4d5fa878fca8bf038b18; no voters:
I20260805 23:09:15.745394 30294 raft_consensus.cc:2781] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:09:15.745491 30294 raft_consensus.cc:482] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:09:15.745522 30294 raft_consensus.cc:3037] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:15.746608 30294 raft_consensus.cc:504] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.746953 30294 leader_election.cc:288] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [CANDIDATE]: Term 1 election: Requested vote from peers 972846a3e3fc4d5fa878fca8bf038b18 (127.3.12.66:38215), e0bc8b0fce694e2fa6d6ca4d8d60ad25 (127.3.12.65:46595)
I20260805 23:09:15.747366 30011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415eac4e6b3b440b81a07e5f941ccaf0" candidate_uuid: "390fa68b41b44552a8d258dd75625f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "972846a3e3fc4d5fa878fca8bf038b18"
I20260805 23:09:15.747453 29843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415eac4e6b3b440b81a07e5f941ccaf0" candidate_uuid: "390fa68b41b44552a8d258dd75625f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25"
I20260805 23:09:15.747490 30011 raft_consensus.cc:3037] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:15.747582 29843 raft_consensus.cc:3037] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:15.748704 30011 raft_consensus.cc:2454] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390fa68b41b44552a8d258dd75625f22 in term 1.
I20260805 23:09:15.748888 29843 raft_consensus.cc:2454] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390fa68b41b44552a8d258dd75625f22 in term 1.
I20260805 23:09:15.748998 30109 leader_election.cc:302] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390fa68b41b44552a8d258dd75625f22, 972846a3e3fc4d5fa878fca8bf038b18; no voters:
I20260805 23:09:15.749300 30294 raft_consensus.cc:2781] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:15.749508 30294 raft_consensus.cc:686] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 1 LEADER]: Becoming Leader. State: Replica: 390fa68b41b44552a8d258dd75625f22, State: Running, Role: LEADER
I20260805 23:09:15.749884 30294 consensus_queue.cc:237] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } }
I20260805 23:09:15.753243 29637 catalog_manager.cc:5495] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 390fa68b41b44552a8d258dd75625f22 (127.3.12.67). New cstate: current_term: 1 leader_uuid: "390fa68b41b44552a8d258dd75625f22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "390fa68b41b44552a8d258dd75625f22" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 33121 } health_report { overall_health: HEALTHY } } }
I20260805 23:09:15.781268 30283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.777619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.3.12.67:33121)
0805 23:09:15.777757 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:15.777761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:15.777798 (+ 37us) server_negotiation.cc:407] Connection header received
0805 23:09:15.777870 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:15.777872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:15.777929 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:15.778022 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:15.778248 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.778752 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.779674 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:15.779903 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:15.779996 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:15.780357 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:15.780870 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:15.780991 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:15.781102 (+ 111us) server_negotiation.cc:299] Negotiation successful
0805 23:09:15.781138 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260805 23:09:15.787604 30011 raft_consensus.cc:1261] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 1 FOLLOWER]: Refusing update from remote peer 390fa68b41b44552a8d258dd75625f22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:15.787636 29843 raft_consensus.cc:1261] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [term 1 FOLLOWER]: Refusing update from remote peer 390fa68b41b44552a8d258dd75625f22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:15.788463 30298 consensus_queue.cc:1035] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "972846a3e3fc4d5fa878fca8bf038b18" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:15.788642 30294 consensus_queue.cc:1035] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0bc8b0fce694e2fa6d6ca4d8d60ad25" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:15.798568 30301 mvcc.cc:205] Tried to move back new op lower bound from 7315338673298284544 to 7315338673154236416. Current Snapshot: MvccSnapshot[applied={T|T < 7315338673298284544}]
I20260805 23:09:15.798557 30303 mvcc.cc:205] Tried to move back new op lower bound from 7315338673298284544 to 7315338673154236416. Current Snapshot: MvccSnapshot[applied={T|T < 7315338673298284544 or (T in {7315338673298284544})}]
I20260805 23:09:15.806567 29643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:15.806720 29643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:15.809505 29643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31387412d8bb434e802b818d62194440: Sending DeleteTablet for 3 replicas of tablet 415eac4e6b3b440b81a07e5f941ccaf0
I20260805 23:09:15.810381 29816 tablet_service.cc:1514] Processing DeleteTablet for tablet 415eac4e6b3b440b81a07e5f941ccaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:15 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:39312
I20260805 23:09:15.810382 30168 tablet_service.cc:1514] Processing DeleteTablet for tablet 415eac4e6b3b440b81a07e5f941ccaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:15 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:55410
I20260805 23:09:15.810945 30314 tablet_replica.cc:330] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25: stopping tablet replica
I20260805 23:09:15.811232 29989 tablet_service.cc:1514] Processing DeleteTablet for tablet 415eac4e6b3b440b81a07e5f941ccaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:15 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:52386
I20260805 23:09:15.811342 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 29728
I20260805 23:09:15.811472 30314 raft_consensus.cc:2229] T 415eac4e6b3b440b81a07e5f941ccaf0 P e0bc8b0fce694e2fa6d6ca4d8d60ad25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:15.812086 30313 tablet_replica.cc:330] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: stopping tablet replica
I20260805 23:09:15.812597 30313 raft_consensus.cc:2229] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:09:15.813033 30313 raft_consensus.cc:2258] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:15.819289 30315 tablet_replica.cc:330] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18: stopping tablet replica
I20260805 23:09:15.819687 30315 raft_consensus.cc:2229] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:15.820202 30315 raft_consensus.cc:2258] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:15.821760 30313 ts_tablet_manager.cc:1905] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:15.825634 30313 ts_tablet_manager.cc:1918] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:15.825809 30313 log.cc:1196] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/415eac4e6b3b440b81a07e5f941ccaf0
I20260805 23:09:15.826277 30313 ts_tablet_manager.cc:1939] T 415eac4e6b3b440b81a07e5f941ccaf0 P 390fa68b41b44552a8d258dd75625f22: Deleting consensus metadata
I20260805 23:09:15.827725 29621 catalog_manager.cc:4841] TS 390fa68b41b44552a8d258dd75625f22 (127.3.12.67:33121): tablet 415eac4e6b3b440b81a07e5f941ccaf0 (table test-table [id=868b09c0dee54f908530f351c552a242]) successfully deleted
I20260805 23:09:15.829358 30315 ts_tablet_manager.cc:1905] T 415eac4e6b3b440b81a07e5f941ccaf0 P 972846a3e3fc4d5fa878fca8bf038b18: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260805 23:09:15.829977 29617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:46595 (error 108)
I20260805 23:09:15.830298 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 29914
I20260805 23:09:15.839462 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.839143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:46595 (local address 127.0.0.1:39322)
0805 23:09:15.839313 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.839374 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:46595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260805 23:09:15.839628 29617 catalog_manager.cc:4584] TS e0bc8b0fce694e2fa6d6ca4d8d60ad25 (127.3.12.65:46595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 415eac4e6b3b440b81a07e5f941ccaf0: Network error: Client connection negotiation failed: client connection to 127.3.12.65:46595: connect: Connection refused (error 111)
I20260805 23:09:15.846884 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 30085
I20260805 23:09:15.847067 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:15.846746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38215 (local address 127.0.0.1:52402)
0805 23:09:15.846932 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:15.846983 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.66:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260805 23:09:15.869436 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 29592
2026-08-05T23:09:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2622 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-05T23:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:15Z Disabled control of system clock
I20260805 23:09:15.908895 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42531
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 23:09:16.021281 30320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:16.021543 30320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:16.021579 30320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:16.025490 30320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:16.025547 30320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:16.025563 30320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:16.025579 30320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:16.025594 30320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:16.030273 30320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:42531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:16.031335 30320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:16.032718 30320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:16.041751 30327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:16.041777 30328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:16.041834 30320 server_base.cc:1034] running on GCE node
W20260805 23:09:16.041778 30330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:16.042609 30320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:16.043730 30320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:16.044901 30320 hybrid_clock.cc:648] HybridClock initialized: now 1785971356044886 us; error 28 us; skew 500 ppm
I20260805 23:09:16.046903 30320 webserver.cc:460] Webserver started at http://127.3.12.126:39183/ using document root <none> and password file <none>
I20260805 23:09:16.047545 30320 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:16.047621 30320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:16.047883 30320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:16.050064 30320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "d9e56a69487f47f1b1e79e702159091b"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "42e2887c951f08cddee71bf970391dcf"
server_key_iv: "61614d3201ec9c6c64ca88cd5f6078a2"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.050658 30320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "d9e56a69487f47f1b1e79e702159091b"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "42e2887c951f08cddee71bf970391dcf"
server_key_iv: "61614d3201ec9c6c64ca88cd5f6078a2"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.054484 30320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:16.056985 30337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.058295 30320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:16.058420 30320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "d9e56a69487f47f1b1e79e702159091b"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "42e2887c951f08cddee71bf970391dcf"
server_key_iv: "61614d3201ec9c6c64ca88cd5f6078a2"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.058526 30320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:16.074841 30320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:16.075580 30320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:16.075786 30320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:16.083161 30320 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:42531
I20260805 23:09:16.083209 30404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:42531 every 8 connection(s)
I20260805 23:09:16.084147 30320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:16.085318 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 30320
I20260805 23:09:16.085461 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:16.085806 3121 external_mini_cluster.cc:1428] Setting key 68c8a256bf3522e7f4cd31d35a1337e5
I20260805 23:09:16.088096 30405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:16.094781 30405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b: Bootstrap starting.
I20260805 23:09:16.095758 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.087360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.3.12.126:42531)
0805 23:09:16.088012 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.088028 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.088081 (+ 53us) server_negotiation.cc:407] Connection header received
0805 23:09:16.088930 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.088946 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.089338 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.089626 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:16.089958 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.090919 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.092155 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.092694 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.092811 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.093736 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:16.093741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:16.093754 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:16.094340 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:16.094417 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:16.094441 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:16.094528 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.094696 (+ 168us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.095041 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":153,"threads_started":1}
I20260805 23:09:16.097322 30405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:16.098263 30405 log.cc:824] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:16.100467 30405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b: No bootstrap required, opened a new log
I20260805 23:09:16.103776 30405 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e56a69487f47f1b1e79e702159091b" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42531 } }
I20260805 23:09:16.104031 30405 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:16.104117 30405 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e56a69487f47f1b1e79e702159091b, State: Initialized, Role: FOLLOWER
I20260805 23:09:16.104635 30405 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e56a69487f47f1b1e79e702159091b" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42531 } }
I20260805 23:09:16.104802 30405 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:16.104885 30405 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:16.104964 30405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:16.105800 30405 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e56a69487f47f1b1e79e702159091b" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42531 } }
I20260805 23:09:16.106084 30405 leader_election.cc:302] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9e56a69487f47f1b1e79e702159091b; no voters:
I20260805 23:09:16.106348 30405 leader_election.cc:288] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:16.106508 30414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:16.106793 30414 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [term 1 LEADER]: Becoming Leader. State: Replica: d9e56a69487f47f1b1e79e702159091b, State: Running, Role: LEADER
I20260805 23:09:16.107103 30414 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e56a69487f47f1b1e79e702159091b" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42531 } }
I20260805 23:09:16.107507 30405 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:16.108785 30415 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9e56a69487f47f1b1e79e702159091b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e56a69487f47f1b1e79e702159091b" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42531 } } }
I20260805 23:09:16.108922 30415 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:16.108773 30417 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9e56a69487f47f1b1e79e702159091b. Latest consensus state: current_term: 1 leader_uuid: "d9e56a69487f47f1b1e79e702159091b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e56a69487f47f1b1e79e702159091b" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42531 } } }
I20260805 23:09:16.109087 30417 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:16.109331 30426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:16.112076 30426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:16.118331 30426 catalog_manager.cc:1347] Generated new cluster ID: 7465d703f4304e85813b9934c2153151
I20260805 23:09:16.118396 30426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:16.130492 30426 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:16.131804 30426 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:16.139353 30426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b: Generated new TSK 0
I20260805 23:09:16.140177 30426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:16.152171 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42531
--builtin_ntp_servers=127.3.12.84:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 23:09:16.268301 30438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:16.268549 30438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:16.268591 30438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:16.272594 30438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:16.272647 30438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:16.272730 30438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:09:16.277420 30438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:16.278541 30438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:16.279985 30438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:16.290195 30452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:16.290264 30448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:16.290285 30450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:16.290441 30438 server_base.cc:1034] running on GCE node
I20260805 23:09:16.290915 30438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:16.291571 30438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:16.292755 30438 hybrid_clock.cc:648] HybridClock initialized: now 1785971356292734 us; error 42 us; skew 500 ppm
I20260805 23:09:16.294597 30438 webserver.cc:460] Webserver started at http://127.3.12.65:33085/ using document root <none> and password file <none>
I20260805 23:09:16.295190 30438 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:16.295282 30438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:16.295544 30438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:16.297775 30438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "167fd22f56654a22a7318926650fc0cf"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "bc9904d5258c868247c7eba903648da8"
server_key_iv: "099ceab62f65b1673fc89d7001a1cea8"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.298375 30438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "167fd22f56654a22a7318926650fc0cf"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "bc9904d5258c868247c7eba903648da8"
server_key_iv: "099ceab62f65b1673fc89d7001a1cea8"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.302172 30438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:16.304672 30459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.305894 30438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:16.306034 30438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "167fd22f56654a22a7318926650fc0cf"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "bc9904d5258c868247c7eba903648da8"
server_key_iv: "099ceab62f65b1673fc89d7001a1cea8"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.306156 30438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:16.319335 30438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:16.320019 30438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:16.320222 30438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:16.320839 30438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:16.321802 30438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:16.321875 30438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.321945 30438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:16.322000 30438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.331248 30438 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:34977
I20260805 23:09:16.331271 30625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:34977 every 8 connection(s)
I20260805 23:09:16.332160 30438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:09:16.340091 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 30438
I20260805 23:09:16.340246 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:09:16.340610 3121 external_mini_cluster.cc:1428] Setting key 96b32eff0fa6aca86dedc183294ea782
I20260805 23:09:16.341609 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.334019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:34201 (local address 127.3.12.126:42531)
0805 23:09:16.334176 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.334179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.335231 (+ 1052us) server_negotiation.cc:407] Connection header received
0805 23:09:16.336698 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.336702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.336765 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.336878 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:16.337770 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.338308 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.339476 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.339707 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.339822 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.341069 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:16.341073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:16.341078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:16.341096 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:16.341116 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:16.341124 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:16.341208 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.341372 (+ 164us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.341463 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260805 23:09:16.342478 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.334428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42531 (local address 127.3.12.65:34201)
0805 23:09:16.335074 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:16.335113 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:16.336475 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:16.337054 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:16.337060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:16.337142 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:16.337626 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.337639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.338474 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.338477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.339341 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.339351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.339866 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.339870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.340034 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.340773 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:16.340777 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:16.340824 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:16.340888 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:16.341210 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:16.341223 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:16.341230 (+ 7us) client_negotiation.cc:769] Sending connection context
0805 23:09:16.341377 (+ 147us) client_negotiation.cc:240] Negotiation successful
0805 23:09:16.341671 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260805 23:09:16.342636 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42531
--builtin_ntp_servers=127.3.12.84:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 23:09:16.343693 30626 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42531
I20260805 23:09:16.343940 30626 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:16.344607 30626 heartbeater.cc:511] Master 127.3.12.126:42531 requested a full tablet report, sending...
I20260805 23:09:16.346190 30360 ts_manager.cc:194] Registered new tserver with Master: 167fd22f56654a22a7318926650fc0cf (127.3.12.65:34977)
I20260805 23:09:16.347465 30360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.65:34201
W20260805 23:09:16.459736 30630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:16.460000 30630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:16.460039 30630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:16.464218 30630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:16.464290 30630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:16.464349 30630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:09:16.469098 30630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:16.470227 30630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:16.471640 30630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:16.481964 30639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:16.482039 30636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:16.482196 30630 server_base.cc:1034] running on GCE node
W20260805 23:09:16.482045 30637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:16.482808 30630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:16.483395 30630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:16.484557 30630 hybrid_clock.cc:648] HybridClock initialized: now 1785971356484530 us; error 39 us; skew 500 ppm
I20260805 23:09:16.486543 30630 webserver.cc:460] Webserver started at http://127.3.12.66:43119/ using document root <none> and password file <none>
I20260805 23:09:16.487140 30630 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:16.487206 30630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:16.487421 30630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:16.489542 30630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "476368590f574a95bd779afa17efbf07"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "2f0c4502dffcb7f91187ae68c9b9ab73"
server_key_iv: "88e7f179de23865df40f3dd6248c2270"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.490134 30630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "476368590f574a95bd779afa17efbf07"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "2f0c4502dffcb7f91187ae68c9b9ab73"
server_key_iv: "88e7f179de23865df40f3dd6248c2270"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.493929 30630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260805 23:09:16.496392 30647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.497635 30630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:09:16.497768 30630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "476368590f574a95bd779afa17efbf07"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "2f0c4502dffcb7f91187ae68c9b9ab73"
server_key_iv: "88e7f179de23865df40f3dd6248c2270"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.497875 30630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:16.519191 30630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:16.519927 30630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:16.520147 30630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:16.520824 30630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:16.521867 30630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:16.521920 30630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.521979 30630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:16.521996 30630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.532289 30630 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:38407
I20260805 23:09:16.532305 30798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:38407 every 8 connection(s)
I20260805 23:09:16.533272 30630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:09:16.533605 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 30630
I20260805 23:09:16.533747 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:09:16.534137 3121 external_mini_cluster.cc:1428] Setting key 05266f28f5d69dd33bad8442e3938159
I20260805 23:09:16.536639 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42531
--builtin_ntp_servers=127.3.12.84:45823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 23:09:16.545002 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.535372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:51465 (local address 127.3.12.126:42531)
0805 23:09:16.535531 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.535535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.536930 (+ 1395us) server_negotiation.cc:407] Connection header received
0805 23:09:16.538522 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.538525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.538572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.538650 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:16.539935 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.540558 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.542554 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.542776 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.542874 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.544399 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:16.544402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:16.544404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:16.544418 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:16.544439 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:16.544447 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:16.544513 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.544765 (+ 252us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.544830 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 23:09:16.546298 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.535925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42531 (local address 127.3.12.66:51465)
0805 23:09:16.536752 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:16.536831 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:16.538245 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:16.538884 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:16.538895 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:16.539020 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:16.539757 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.539780 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.540707 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.540714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.542373 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.542389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.542981 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.542987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.543253 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.544029 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:16.544032 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:16.544098 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:16.544199 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:16.544548 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:16.544561 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:16.544571 (+ 10us) client_negotiation.cc:769] Sending connection context
0805 23:09:16.544726 (+ 155us) client_negotiation.cc:240] Negotiation successful
0805 23:09:16.545104 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":150,"threads_started":1}
I20260805 23:09:16.547688 30799 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42531
I20260805 23:09:16.547947 30799 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:16.548621 30799 heartbeater.cc:511] Master 127.3.12.126:42531 requested a full tablet report, sending...
I20260805 23:09:16.549755 30360 ts_manager.cc:194] Registered new tserver with Master: 476368590f574a95bd779afa17efbf07 (127.3.12.66:38407)
I20260805 23:09:16.550338 30360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.66:51465
W20260805 23:09:16.659439 30802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:16.659742 30802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:16.659816 30802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:16.663971 30802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:16.664072 30802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:16.664163 30802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:09:16.669179 30802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:45823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:42531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:16.670416 30802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:16.671855 30802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:16.682000 30809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:16.681947 30810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:16.681972 30812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:16.682219 30802 server_base.cc:1034] running on GCE node
I20260805 23:09:16.682603 30802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:16.683257 30802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:16.684432 30802 hybrid_clock.cc:648] HybridClock initialized: now 1785971356684419 us; error 32 us; skew 500 ppm
I20260805 23:09:16.686489 30802 webserver.cc:460] Webserver started at http://127.3.12.67:41181/ using document root <none> and password file <none>
I20260805 23:09:16.687120 30802 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:16.687186 30802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:16.687400 30802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:16.689575 30802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "51a7480917774aa2aa2c472c3af0d6d0"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "5746dd1bb087e8f0e9e544d0dfd1f991"
server_key_iv: "1a20c69fc26a462e58d410097daa0705"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.690182 30802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "51a7480917774aa2aa2c472c3af0d6d0"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "5746dd1bb087e8f0e9e544d0dfd1f991"
server_key_iv: "1a20c69fc26a462e58d410097daa0705"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.694074 30802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260805 23:09:16.696605 30819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.697774 30802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:09:16.697916 30802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "51a7480917774aa2aa2c472c3af0d6d0"
format_stamp: "Formatted at 2026-08-05 23:09:16 on dist-test-slave-2d19"
server_key: "5746dd1bb087e8f0e9e544d0dfd1f991"
server_key_iv: "1a20c69fc26a462e58d410097daa0705"
server_key_version: "encryptionkey@0"
I20260805 23:09:16.698032 30802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:16.727597 30802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:16.728404 30802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:16.728624 30802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:16.729274 30802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:16.730273 30802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:16.730347 30802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.730424 30802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:16.730470 30802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:16.739840 30802 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:46257
I20260805 23:09:16.739871 30972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:46257 every 8 connection(s)
I20260805 23:09:16.740768 30802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:09:16.744870 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 30802
I20260805 23:09:16.745002 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:09:16.745288 3121 external_mini_cluster.cc:1428] Setting key 7d6cf7319aadc2dac3cf6efaf5fbd3bb
I20260805 23:09:16.750401 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.742716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:41621 (local address 127.3.12.126:42531)
0805 23:09:16.742866 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.742870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.743920 (+ 1050us) server_negotiation.cc:407] Connection header received
0805 23:09:16.745421 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.745424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.745487 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.745604 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:16.746774 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.747378 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.748566 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.748802 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.748887 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.750005 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:16.750008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:16.750010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:16.750022 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:16.750042 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:16.750050 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:16.750111 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.750240 (+ 129us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.750291 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260805 23:09:16.751255 30976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.743097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:42531 (local address 127.3.12.67:41621)
0805 23:09:16.743766 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:16.743807 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:16.745192 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:16.745818 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:16.745827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:16.745938 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:16.746584 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.746603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.747474 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.747477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.748431 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.748441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.748934 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.748937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.749089 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.749745 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:16.749747 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:16.749785 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:16.749849 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:16.750118 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:16.750123 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:16.750127 (+ 4us) client_negotiation.cc:769] Sending connection context
0805 23:09:16.750214 (+ 87us) client_negotiation.cc:240] Negotiation successful
0805 23:09:16.750428 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260805 23:09:16.752336 30974 heartbeater.cc:347] Connected to a master server at 127.3.12.126:42531
I20260805 23:09:16.752569 30974 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:16.753105 30974 heartbeater.cc:511] Master 127.3.12.126:42531 requested a full tablet report, sending...
I20260805 23:09:16.754232 30360 ts_manager.cc:194] Registered new tserver with Master: 51a7480917774aa2aa2c472c3af0d6d0 (127.3.12.67:46257)
I20260805 23:09:16.754781 30360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.67:41621
I20260805 23:09:16.759231 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:09:16.765259 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.761324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.3.12.126:42531)
0805 23:09:16.761448 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.761451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.761663 (+ 212us) server_negotiation.cc:407] Connection header received
0805 23:09:16.761761 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.761763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.761802 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.761871 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:16.762182 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.762730 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.763655 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.763861 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.763941 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.764770 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:16.764773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:16.764775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:16.764786 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:16.764806 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:16.764813 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:16.764917 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.765112 (+ 195us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.765148 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260805 23:09:16.768680 30360 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:09:16.771843 30360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:09:16.786717 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.780604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:46257 (local address 127.0.0.1:56022)
0805 23:09:16.781971 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:16.781985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:16.782081 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:16.782683 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:16.782686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:16.782704 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:16.782793 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.782801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.784150 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.784154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.785299 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.785309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.786214 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.786218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.786330 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.786353 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:09:16.786420 (+ 67us) client_negotiation.cc:240] Negotiation successful
0805 23:09:16.786549 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":5,"thread_start_us":120,"threads_started":1}
I20260805 23:09:16.786722 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.781441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.3.12.67:46257)
0805 23:09:16.781939 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.781951 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.782185 (+ 234us) server_negotiation.cc:407] Connection header received
0805 23:09:16.782273 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.782279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.782538 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.782710 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:16.782891 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.783994 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.785407 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.786081 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.786212 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.786399 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.786452 (+ 53us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.786549 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":164,"threads_started":1}
I20260805 23:09:16.787283 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.780643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.3.12.65:34977)
0805 23:09:16.781193 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.781212 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.781241 (+ 29us) server_negotiation.cc:407] Connection header received
0805 23:09:16.781321 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.781326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.781561 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.781733 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:16.782284 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.783394 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.784958 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.785683 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.785816 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.786030 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.786749 (+ 719us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.787108 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":100,"threads_started":1}
I20260805 23:09:16.787369 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.781128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:38407 (local address 127.0.0.1:50112)
0805 23:09:16.782088 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:16.782103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:16.782196 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:16.782969 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:16.782972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:16.782990 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:16.783071 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.783076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.784489 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.784495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.785653 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.785664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.786962 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.786966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.787057 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.787079 (+ 22us) client_negotiation.cc:769] Sending connection context
0805 23:09:16.787158 (+ 79us) client_negotiation.cc:240] Negotiation successful
0805 23:09:16.787231 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":47,"thread_start_us":96,"threads_started":1}
I20260805 23:09:16.787102 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.780382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:34977 (local address 127.0.0.1:60498)
0805 23:09:16.780882 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:16.780946 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:16.781081 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:16.781936 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:16.781943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:16.781983 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:16.782139 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.782150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.783521 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.783528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.784804 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.784815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.786098 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.786101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.786208 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.786291 (+ 83us) client_negotiation.cc:769] Sending connection context
0805 23:09:16.786403 (+ 112us) client_negotiation.cc:240] Negotiation successful
0805 23:09:16.787000 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":565,"thread_start_us":94,"threads_started":1}
I20260805 23:09:16.788549 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.781364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.3.12.66:38407)
0805 23:09:16.781740 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.781752 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.782306 (+ 554us) server_negotiation.cc:407] Connection header received
0805 23:09:16.782443 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.782448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.782647 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.782811 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:16.783215 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.784346 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.785803 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.786615 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.786722 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.786897 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.787225 (+ 328us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.787331 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":159,"threads_started":1}
I20260805 23:09:16.790154 30878 tablet_service.cc:1467] Processing CreateTablet for tablet b4efba6ffc2b46c8af2695e90cea3f2e (DEFAULT_TABLE table=test-table [id=c7f29f063e6b4c34b24e8edc262abc76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:16.790460 30527 tablet_service.cc:1467] Processing CreateTablet for tablet b4efba6ffc2b46c8af2695e90cea3f2e (DEFAULT_TABLE table=test-table [id=c7f29f063e6b4c34b24e8edc262abc76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:16.791235 30878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4efba6ffc2b46c8af2695e90cea3f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:16.790979 30704 tablet_service.cc:1467] Processing CreateTablet for tablet b4efba6ffc2b46c8af2695e90cea3f2e (DEFAULT_TABLE table=test-table [id=c7f29f063e6b4c34b24e8edc262abc76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:16.791491 30527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4efba6ffc2b46c8af2695e90cea3f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:16.791960 30704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4efba6ffc2b46c8af2695e90cea3f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:16.797883 30995 tablet_bootstrap.cc:492] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Bootstrap starting.
I20260805 23:09:16.799590 30997 tablet_bootstrap.cc:492] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: Bootstrap starting.
I20260805 23:09:16.800132 30995 tablet_bootstrap.cc:654] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:16.800625 30996 tablet_bootstrap.cc:492] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf: Bootstrap starting.
I20260805 23:09:16.801184 30995 log.cc:824] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:16.801633 30997 tablet_bootstrap.cc:654] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:16.802496 30997 log.cc:824] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:16.802809 30996 tablet_bootstrap.cc:654] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:16.803542 30995 tablet_bootstrap.cc:492] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: No bootstrap required, opened a new log
I20260805 23:09:16.803612 30996 log.cc:824] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:16.803748 30995 ts_tablet_manager.cc:1397] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:09:16.804549 30997 tablet_bootstrap.cc:492] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: No bootstrap required, opened a new log
I20260805 23:09:16.804731 30997 ts_tablet_manager.cc:1397] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260805 23:09:16.805835 30996 tablet_bootstrap.cc:492] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf: No bootstrap required, opened a new log
I20260805 23:09:16.806026 30996 ts_tablet_manager.cc:1397] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:09:16.808212 30997 raft_consensus.cc:348] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.808594 30997 raft_consensus.cc:374] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:16.808660 30997 raft_consensus.cc:729] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476368590f574a95bd779afa17efbf07, State: Initialized, Role: FOLLOWER
I20260805 23:09:16.808683 30995 raft_consensus.cc:348] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.808944 30995 raft_consensus.cc:374] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:16.809001 30995 raft_consensus.cc:729] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a7480917774aa2aa2c472c3af0d6d0, State: Initialized, Role: FOLLOWER
I20260805 23:09:16.809165 30997 consensus_queue.cc:260] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.809489 30995 consensus_queue.cc:260] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.809958 30799 heartbeater.cc:503] Master 127.3.12.126:42531 was elected leader, sending a full tablet report...
I20260805 23:09:16.810331 30996 raft_consensus.cc:348] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.810568 30995 ts_tablet_manager.cc:1428] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260805 23:09:16.810590 30996 raft_consensus.cc:374] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:16.810648 30996 raft_consensus.cc:729] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167fd22f56654a22a7318926650fc0cf, State: Initialized, Role: FOLLOWER
I20260805 23:09:16.810901 30974 heartbeater.cc:503] Master 127.3.12.126:42531 was elected leader, sending a full tablet report...
I20260805 23:09:16.811103 30996 consensus_queue.cc:260] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.811793 30997 ts_tablet_manager.cc:1428] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260805 23:09:16.812490 30626 heartbeater.cc:503] Master 127.3.12.126:42531 was elected leader, sending a full tablet report...
I20260805 23:09:16.813138 30996 ts_tablet_manager.cc:1428] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260805 23:09:16.833381 30628 tablet.cc:2369] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:09:16.935853 31002 raft_consensus.cc:482] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:09:16.936077 31002 raft_consensus.cc:504] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.937376 31002 leader_election.cc:288] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 167fd22f56654a22a7318926650fc0cf (127.3.12.65:34977), 51a7480917774aa2aa2c472c3af0d6d0 (127.3.12.67:46257)
I20260805 23:09:16.942313 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.937561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:34977 (local address 127.3.12.66:43267)
0805 23:09:16.937751 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:16.937769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:16.937865 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:16.938164 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:16.938167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:16.938192 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:16.938337 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.938345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.939403 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.939405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.940681 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.940692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.941782 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.941785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.941912 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.941957 (+ 45us) client_negotiation.cc:769] Sending connection context
0805 23:09:16.942042 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 23:09:16.942149 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":35}
I20260805 23:09:16.942366 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.937711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:43267 (local address 127.3.12.65:34977)
0805 23:09:16.937852 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.937859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.937890 (+ 31us) server_negotiation.cc:407] Connection header received
0805 23:09:16.937952 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.937955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.938040 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.938114 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:16.938487 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.939279 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.940839 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.941622 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.941749 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.941813 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.942129 (+ 316us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.942226 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:09:16.942366 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.937883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:43599 (local address 127.3.12.67:46257)
0805 23:09:16.938028 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.938032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.938277 (+ 245us) server_negotiation.cc:407] Connection header received
0805 23:09:16.938360 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.938362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.938432 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.938517 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:16.938787 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.939524 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.941039 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.941651 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.941749 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.941813 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.942129 (+ 316us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.942224 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:09:16.942322 31008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.937706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:46257 (local address 127.3.12.66:43599)
0805 23:09:16.938164 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:16.938177 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:16.938272 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:16.938543 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:16.938545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:16.938561 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:16.938665 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.938669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.939653 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.939656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.940812 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:16.940820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.941781 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.941785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:16.941922 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.941963 (+ 41us) client_negotiation.cc:769] Sending connection context
0805 23:09:16.942042 (+ 79us) client_negotiation.cc:240] Negotiation successful
0805 23:09:16.942142 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":56,"thread_start_us":148,"threads_started":1}
I20260805 23:09:16.943156 30554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4efba6ffc2b46c8af2695e90cea3f2e" candidate_uuid: "476368590f574a95bd779afa17efbf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167fd22f56654a22a7318926650fc0cf" is_pre_election: true
I20260805 23:09:16.943539 30554 raft_consensus.cc:2454] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476368590f574a95bd779afa17efbf07 in term 0.
I20260805 23:09:16.943531 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4efba6ffc2b46c8af2695e90cea3f2e" candidate_uuid: "476368590f574a95bd779afa17efbf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a7480917774aa2aa2c472c3af0d6d0" is_pre_election: true
I20260805 23:09:16.943826 30912 raft_consensus.cc:2454] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476368590f574a95bd779afa17efbf07 in term 0.
I20260805 23:09:16.944084 30653 leader_election.cc:302] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167fd22f56654a22a7318926650fc0cf, 476368590f574a95bd779afa17efbf07; no voters:
I20260805 23:09:16.944473 31002 raft_consensus.cc:2781] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:09:16.944566 31002 raft_consensus.cc:482] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:09:16.944610 31002 raft_consensus.cc:3037] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:16.945679 31002 raft_consensus.cc:504] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.946069 31002 leader_election.cc:288] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [CANDIDATE]: Term 1 election: Requested vote from peers 167fd22f56654a22a7318926650fc0cf (127.3.12.65:34977), 51a7480917774aa2aa2c472c3af0d6d0 (127.3.12.67:46257)
I20260805 23:09:16.946424 30554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4efba6ffc2b46c8af2695e90cea3f2e" candidate_uuid: "476368590f574a95bd779afa17efbf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167fd22f56654a22a7318926650fc0cf"
I20260805 23:09:16.946487 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4efba6ffc2b46c8af2695e90cea3f2e" candidate_uuid: "476368590f574a95bd779afa17efbf07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a7480917774aa2aa2c472c3af0d6d0"
I20260805 23:09:16.946599 30554 raft_consensus.cc:3037] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:16.946647 30912 raft_consensus.cc:3037] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:16.947676 30554 raft_consensus.cc:2454] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476368590f574a95bd779afa17efbf07 in term 1.
I20260805 23:09:16.947801 30912 raft_consensus.cc:2454] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476368590f574a95bd779afa17efbf07 in term 1.
I20260805 23:09:16.948026 30653 leader_election.cc:302] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167fd22f56654a22a7318926650fc0cf, 476368590f574a95bd779afa17efbf07; no voters:
I20260805 23:09:16.948235 31002 raft_consensus.cc:2781] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:16.948518 31002 raft_consensus.cc:686] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 1 LEADER]: Becoming Leader. State: Replica: 476368590f574a95bd779afa17efbf07, State: Running, Role: LEADER
I20260805 23:09:16.948861 31002 consensus_queue.cc:237] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } }
I20260805 23:09:16.952466 30359 catalog_manager.cc:5495] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 476368590f574a95bd779afa17efbf07 (127.3.12.66). New cstate: current_term: 1 leader_uuid: "476368590f574a95bd779afa17efbf07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476368590f574a95bd779afa17efbf07" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 38407 } health_report { overall_health: HEALTHY } } }
I20260805 23:09:16.981680 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:16.977915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.3.12.66:38407)
0805 23:09:16.978080 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:16.978084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:16.978099 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:09:16.978214 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:16.978216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:16.978276 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:16.978368 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:16.978605 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.979128 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.980026 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:16.980312 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:16.980406 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:16.980710 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:16.981138 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:16.981253 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:16.981431 (+ 178us) server_negotiation.cc:299] Negotiation successful
0805 23:09:16.981491 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 23:09:16.988114 30912 raft_consensus.cc:1261] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 1 FOLLOWER]: Refusing update from remote peer 476368590f574a95bd779afa17efbf07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:16.988114 30554 raft_consensus.cc:1261] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 1 FOLLOWER]: Refusing update from remote peer 476368590f574a95bd779afa17efbf07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:16.989215 31002 consensus_queue.cc:1035] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "167fd22f56654a22a7318926650fc0cf" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:16.989532 31009 consensus_queue.cc:1035] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a7480917774aa2aa2c472c3af0d6d0" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 23:09:16.991833 30975 tablet.cc:2369] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:09:16.996575 31013 mvcc.cc:205] Tried to move back new op lower bound from 7315338678214311936 to 7315338678065508352. Current Snapshot: MvccSnapshot[applied={T|T < 7315338678214311936}]
I20260805 23:09:16.998428 31018 mvcc.cc:205] Tried to move back new op lower bound from 7315338678214311936 to 7315338678065508352. Current Snapshot: MvccSnapshot[applied={T|T < 7315338678214311936}]
I20260805 23:09:17.007071 30359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:17.007277 30359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:17.010180 30359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9e56a69487f47f1b1e79e702159091b: Sending DeleteTablet for 3 replicas of tablet b4efba6ffc2b46c8af2695e90cea3f2e
I20260805 23:09:17.011056 30527 tablet_service.cc:1514] Processing DeleteTablet for tablet b4efba6ffc2b46c8af2695e90cea3f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:17 UTC) from {username='slave'} at 127.0.0.1:60498
I20260805 23:09:17.011415 30704 tablet_service.cc:1514] Processing DeleteTablet for tablet b4efba6ffc2b46c8af2695e90cea3f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:17 UTC) from {username='slave'} at 127.0.0.1:50112
I20260805 23:09:17.011727 31027 tablet_replica.cc:330] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf: stopping tablet replica
I20260805 23:09:17.011670 30878 tablet_service.cc:1514] Processing DeleteTablet for tablet b4efba6ffc2b46c8af2695e90cea3f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:17 UTC) from {username='slave'} at 127.0.0.1:56022
I20260805 23:09:17.012174 31032 tablet_replica.cc:330] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: stopping tablet replica
I20260805 23:09:17.012174 31027 raft_consensus.cc:2229] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:17.012511 31032 raft_consensus.cc:2229] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:17.012748 31027 raft_consensus.cc:2258] T b4efba6ffc2b46c8af2695e90cea3f2e P 167fd22f56654a22a7318926650fc0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:17.012784 31032 raft_consensus.cc:2258] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:17.012912 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 30438
I20260805 23:09:17.033034 31032 ts_tablet_manager.cc:1905] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:17.036496 31032 ts_tablet_manager.cc:1918] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:17.036614 31032 log.cc:1196] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/b4efba6ffc2b46c8af2695e90cea3f2e
I20260805 23:09:17.036922 31032 ts_tablet_manager.cc:1939] T b4efba6ffc2b46c8af2695e90cea3f2e P 51a7480917774aa2aa2c472c3af0d6d0: Deleting consensus metadata
I20260805 23:09:17.038357 30342 catalog_manager.cc:4841] TS 51a7480917774aa2aa2c472c3af0d6d0 (127.3.12.67:46257): tablet b4efba6ffc2b46c8af2695e90cea3f2e (table test-table [id=c7f29f063e6b4c34b24e8edc262abc76]) successfully deleted
I20260805 23:09:17.047997 31031 tablet_replica.cc:330] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: stopping tablet replica
I20260805 23:09:17.048427 31031 raft_consensus.cc:2229] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:09:17.048894 31031 raft_consensus.cc:2258] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:17.050384 31031 ts_tablet_manager.cc:1905] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:17.052860 31031 ts_tablet_manager.cc:1918] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:17.052996 31031 log.cc:1196] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/b4efba6ffc2b46c8af2695e90cea3f2e
I20260805 23:09:17.053304 31031 ts_tablet_manager.cc:1939] T b4efba6ffc2b46c8af2695e90cea3f2e P 476368590f574a95bd779afa17efbf07: Deleting consensus metadata
I20260805 23:09:17.054544 30342 catalog_manager.cc:4841] TS 476368590f574a95bd779afa17efbf07 (127.3.12.66:38407): tablet b4efba6ffc2b46c8af2695e90cea3f2e (table test-table [id=c7f29f063e6b4c34b24e8edc262abc76]) successfully deleted
W20260805 23:09:17.055020 30344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:34977 (error 108)
I20260805 23:09:17.055286 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 30630
I20260805 23:09:17.061241 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:17.060934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:34977 (local address 127.0.0.1:60504)
0805 23:09:17.061071 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:17.061156 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260805 23:09:17.061475 30344 catalog_manager.cc:4584] TS 167fd22f56654a22a7318926650fc0cf (127.3.12.65:34977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4efba6ffc2b46c8af2695e90cea3f2e: Network error: Client connection negotiation failed: client connection to 127.3.12.65:34977: connect: Connection refused (error 111)
I20260805 23:09:17.067556 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 30802
I20260805 23:09:17.078243 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 30320
2026-08-05T23:09:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1206 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:17 dist-test-slave-2d19 krb5kdc[31037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:17 dist-test-slave-2d19 krb5kdc[31037](info): set up 2 sockets
Aug 05 23:09:17 dist-test-slave-2d19 krb5kdc[31037](info): commencing operation
krb5kdc: starting...
W20260805 23:09:18.318418 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:18 dist-test-slave-2d19 krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:18.452327 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40251
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:18.567754 31057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:18.568056 31057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:18.568097 31057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:18.572294 31057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:18.572367 31057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:18.572387 31057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:18.572405 31057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:18.572422 31057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:18.577898 31057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:40251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:18.579088 31057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:18.580222 31057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:18.589407 31065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:18.589475 31057 server_base.cc:1034] running on GCE node
W20260805 23:09:18.589390 31069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:18.589390 31067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:18.590246 31057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:18.591394 31057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:18.592592 31057 hybrid_clock.cc:648] HybridClock initialized: now 1785971358592559 us; error 47 us; skew 500 ppm
Aug 05 23:09:18 dist-test-slave-2d19 krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:18.595571 31057 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:18.596566 31057 webserver.cc:460] Webserver started at http://127.3.12.126:42435/ using document root <none> and password file <none>
I20260805 23:09:18.597149 31057 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:18.597245 31057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:18.597489 31057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:18.599654 31057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae"
format_stamp: "Formatted at 2026-08-05 23:09:18 on dist-test-slave-2d19"
server_key: "36e7ad9d6b2c9f514a4e0243b2669727"
server_key_iv: "a39843501d73b66d63baf8ff7114aea5"
server_key_version: "encryptionkey@0"
I20260805 23:09:18.600330 31057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae"
format_stamp: "Formatted at 2026-08-05 23:09:18 on dist-test-slave-2d19"
server_key: "36e7ad9d6b2c9f514a4e0243b2669727"
server_key_iv: "a39843501d73b66d63baf8ff7114aea5"
server_key_version: "encryptionkey@0"
I20260805 23:09:18.604172 31057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260805 23:09:18.606642 31078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:18.607918 31057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:18.608095 31057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae"
format_stamp: "Formatted at 2026-08-05 23:09:18 on dist-test-slave-2d19"
server_key: "36e7ad9d6b2c9f514a4e0243b2669727"
server_key_iv: "a39843501d73b66d63baf8ff7114aea5"
server_key_version: "encryptionkey@0"
I20260805 23:09:18.608222 31057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:18.622200 31057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:18.625959 31057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:18.626204 31057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:18.633688 31057 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:40251
I20260805 23:09:18.633754 31152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:40251 every 8 connection(s)
I20260805 23:09:18.634737 31057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:18.637688 31156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:18.639832 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 31057
I20260805 23:09:18.639994 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:18.640378 3121 external_mini_cluster.cc:1428] Setting key 1ccd87b74106b57b60642869984cbd0d
I20260805 23:09:18.644544 31156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae: Bootstrap starting.
I20260805 23:09:18.647378 31156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:18.648316 31156 log.cc:824] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:09:18 dist-test-slave-2d19 krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:18.650810 31156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae: No bootstrap required, opened a new log
I20260805 23:09:18.655777 31156 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40251 } }
I20260805 23:09:18.656078 31156 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:18.656152 31156 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f7a027cdd64dbaa75bfabcdb0fc0ae, State: Initialized, Role: FOLLOWER
I20260805 23:09:18.656689 31156 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40251 } }
I20260805 23:09:18.656854 31156 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:18.656924 31156 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:18.656981 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:18.641939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.3.12.126:40251)
0805 23:09:18.642414 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:18.642436 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:18.642483 (+ 47us) server_negotiation.cc:407] Connection header received
0805 23:09:18.643358 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:18.643382 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:18.643811 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:18.644191 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:18.645016 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:18.646520 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:18.647413 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:18.648106 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:18.648310 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:18.651310 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:18.651317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:18.651335 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:18.651369 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:18.653555 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:18.654022 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:18.654028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:18.654034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:18.654193 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:18.654542 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:18.654545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:18.654547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:18.655098 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:18.655314 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:18.655593 (+ 279us) server_negotiation.cc:299] Negotiation successful
0805 23:09:18.655905 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":114,"threads_started":1}
I20260805 23:09:18.657029 31156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:18.658053 31156 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40251 } }
I20260805 23:09:18.658427 31156 leader_election.cc:302] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0f7a027cdd64dbaa75bfabcdb0fc0ae; no voters:
I20260805 23:09:18.658812 31156 leader_election.cc:288] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:18.658910 31167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:18.659093 31167 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [term 1 LEADER]: Becoming Leader. State: Replica: a0f7a027cdd64dbaa75bfabcdb0fc0ae, State: Running, Role: LEADER
I20260805 23:09:18.659401 31167 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40251 } }
I20260805 23:09:18.660177 31156 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:18.660616 31169 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40251 } } }
I20260805 23:09:18.660748 31169 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:18.661003 31170 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0f7a027cdd64dbaa75bfabcdb0fc0ae. Latest consensus state: current_term: 1 leader_uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f7a027cdd64dbaa75bfabcdb0fc0ae" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40251 } } }
I20260805 23:09:18.661111 31170 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:18.661201 31172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:18.664559 31172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:18.670359 31172 catalog_manager.cc:1347] Generated new cluster ID: 60255a0a3fb1461986dd70495d29f158
I20260805 23:09:18.670442 31172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:18.685570 31172 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:18.686825 31172 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:18.693638 31172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae: Generated new TSK 0
I20260805 23:09:18.694415 31172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
Entry for principal kudu/127.3.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab.
I20260805 23:09:18.739161 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40251
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:18.875535 31192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:18.875808 31192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:18.875852 31192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:18.879923 31192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:18.879992 31192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:18.880051 31192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:09:18.885396 31192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.65.keytab
--principal=kudu/127.3.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:18.886498 31192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:18.887549 31192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:18.897676 31201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:18.897692 31200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:18.897673 31203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:18.897938 31192 server_base.cc:1034] running on GCE node
I20260805 23:09:18.898343 31192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:18.899030 31192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:18.900203 31192 hybrid_clock.cc:648] HybridClock initialized: now 1785971358900181 us; error 33 us; skew 500 ppm
Aug 05 23:09:18 dist-test-slave-2d19 krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:18.903048 31192 init.cc:377] Logged in from keytab as kudu/127.3.12.65@KRBTEST.COM (short username kudu)
I20260805 23:09:18.903972 31192 webserver.cc:460] Webserver started at http://127.3.12.65:32909/ using document root <none> and password file <none>
I20260805 23:09:18.904678 31192 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:18.904783 31192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:18.905001 31192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:18.907160 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "afa2ee279c6b48a69261522a0b45b1ac"
format_stamp: "Formatted at 2026-08-05 23:09:18 on dist-test-slave-2d19"
server_key: "f27177dec4ea257c923d7f784c483943"
server_key_iv: "10da64548839d18eff7555230d2235b7"
server_key_version: "encryptionkey@0"
I20260805 23:09:18.907732 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "afa2ee279c6b48a69261522a0b45b1ac"
format_stamp: "Formatted at 2026-08-05 23:09:18 on dist-test-slave-2d19"
server_key: "f27177dec4ea257c923d7f784c483943"
server_key_iv: "10da64548839d18eff7555230d2235b7"
server_key_version: "encryptionkey@0"
I20260805 23:09:18.911523 31192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:09:18.914177 31218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:18.915393 31192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260805 23:09:18.915546 31192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "afa2ee279c6b48a69261522a0b45b1ac"
format_stamp: "Formatted at 2026-08-05 23:09:18 on dist-test-slave-2d19"
server_key: "f27177dec4ea257c923d7f784c483943"
server_key_iv: "10da64548839d18eff7555230d2235b7"
server_key_version: "encryptionkey@0"
I20260805 23:09:18.915679 31192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:18.928146 31192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:18.931192 31192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:18.931399 31192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:18.931998 31192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:18.932853 31192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:18.932930 31192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:18.933003 31192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:18.933050 31192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:18.942404 31192 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:42453
I20260805 23:09:18.942430 31376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:42453 every 8 connection(s)
I20260805 23:09:18.943292 31192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:09:18.947870 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 31192
I20260805 23:09:18.948066 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:09:18.948421 3121 external_mini_cluster.cc:1428] Setting key d85b5df4eec00f56b817555266621369
Aug 05 23:09:18 dist-test-slave-2d19 krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.65@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:18.959749 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:18.945359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:32991 (local address 127.3.12.126:40251)
0805 23:09:18.945531 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:18.945535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:18.946346 (+ 811us) server_negotiation.cc:407] Connection header received
0805 23:09:18.947776 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:18.947779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:18.947830 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:18.947907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:18.949546 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:18.950260 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:18.952126 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:18.952508 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:18.952636 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:18.955400 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:18.955407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:18.955413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:18.955450 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:18.957376 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:18.957974 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:18.957982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:18.957986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:18.958093 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:18.958461 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:18.958466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:18.958469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:18.958717 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:18.958834 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:18.959399 (+ 565us) server_negotiation.cc:299] Negotiation successful
0805 23:09:18.959544 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.3.12.66@KRBTEST.COM; defaulting to no policy
I20260805 23:09:18.960664 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:18.945728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40251 (local address 127.3.12.65:32991)
0805 23:09:18.946182 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:18.946227 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:18.947493 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:18.948151 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:18.948161 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:18.948727 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:18.949387 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:18.949404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:18.950439 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:18.950444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:18.951969 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:18.951985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:18.952690 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:18.952693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:18.952845 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:18.953532 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:18.953560 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:18.955167 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:18.957524 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:18.957530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:18.957543 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:18.957857 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:18.958199 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:18.958201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:18.958202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:18.958353 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:18.958831 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:18.958837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:18.959137 (+ 300us) client_negotiation.cc:769] Sending connection context
0805 23:09:18.959305 (+ 168us) client_negotiation.cc:240] Negotiation successful
0805 23:09:18.959592 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260805 23:09:18.962234 31379 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40251
I20260805 23:09:18.962531 31379 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:18.963106 31379 heartbeater.cc:511] Master 127.3.12.126:40251 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.66@KRBTEST.COM" created.
I20260805 23:09:18.964934 31100 ts_manager.cc:194] Registered new tserver with Master: afa2ee279c6b48a69261522a0b45b1ac (127.3.12.65:42453)
I20260805 23:09:18.966579 31100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.65@KRBTEST.COM'} at 127.3.12.65:32991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
Entry for principal kudu/127.3.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab.
I20260805 23:09:18.983639 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40251
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:19.100099 31387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:19.100458 31387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:19.100553 31387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:19.104666 31387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:19.104743 31387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:19.104827 31387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:09:19.110196 31387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.66.keytab
--principal=kudu/127.3.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:19.111246 31387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:19.112358 31387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:19.122525 31396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:19.122545 31394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:19.122598 31392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:19.122704 31387 server_base.cc:1034] running on GCE node
I20260805 23:09:19.123180 31387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:19.123872 31387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:19.125041 31387 hybrid_clock.cc:648] HybridClock initialized: now 1785971359124997 us; error 56 us; skew 500 ppm
Aug 05 23:09:19 dist-test-slave-2d19 krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:19.127879 31387 init.cc:377] Logged in from keytab as kudu/127.3.12.66@KRBTEST.COM (short username kudu)
I20260805 23:09:19.128880 31387 webserver.cc:460] Webserver started at http://127.3.12.66:36153/ using document root <none> and password file <none>
I20260805 23:09:19.129456 31387 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:19.129570 31387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:19.129808 31387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:19.131945 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "ba13052606364fa783e04a7c4aa5dd74"
format_stamp: "Formatted at 2026-08-05 23:09:19 on dist-test-slave-2d19"
server_key: "2240ab58dcfb746b1360cc45fb0686b6"
server_key_iv: "21a0bb69bf37084643e449d420e57a1c"
server_key_version: "encryptionkey@0"
I20260805 23:09:19.132566 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "ba13052606364fa783e04a7c4aa5dd74"
format_stamp: "Formatted at 2026-08-05 23:09:19 on dist-test-slave-2d19"
server_key: "2240ab58dcfb746b1360cc45fb0686b6"
server_key_iv: "21a0bb69bf37084643e449d420e57a1c"
server_key_version: "encryptionkey@0"
I20260805 23:09:19.136361 31387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260805 23:09:19.138809 31405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:19.140048 31387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260805 23:09:19.140200 31387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "ba13052606364fa783e04a7c4aa5dd74"
format_stamp: "Formatted at 2026-08-05 23:09:19 on dist-test-slave-2d19"
server_key: "2240ab58dcfb746b1360cc45fb0686b6"
server_key_iv: "21a0bb69bf37084643e449d420e57a1c"
server_key_version: "encryptionkey@0"
I20260805 23:09:19.140360 31387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:19.151726 31387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:19.154815 31387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:19.155028 31387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:19.155624 31387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:19.156692 31387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:19.156765 31387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:19.156833 31387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:19.156867 31387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:19.166852 31387 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:43303
I20260805 23:09:19.166904 31568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:43303 every 8 connection(s)
I20260805 23:09:19.167817 31387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:09:19.171746 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 31387
I20260805 23:09:19.171875 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:09:19.172214 3121 external_mini_cluster.cc:1428] Setting key 086a8172f6d15e41394ae66fd12cac9c
Aug 05 23:09:19 dist-test-slave-2d19 krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.66@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.67@KRBTEST.COM; defaulting to no policy
I20260805 23:09:19.185091 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.169905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:55393 (local address 127.3.12.126:40251)
0805 23:09:19.170093 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.170098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.170928 (+ 830us) server_negotiation.cc:407] Connection header received
0805 23:09:19.172775 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.172778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.172839 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.172926 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:19.174394 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.174921 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.176644 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.176879 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.177018 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.180278 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:19.180319 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:19.180325 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:19.180375 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:19.182351 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:19.183020 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:19.183024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:19.183026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:19.183133 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:19.183562 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:19.183565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:19.183567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:19.183861 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:19.184000 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.184696 (+ 696us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.184853 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260805 23:09:19.185976 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.170295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40251 (local address 127.3.12.66:55393)
0805 23:09:19.170770 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:19.170814 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:19.172506 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:19.173137 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:19.173145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:19.173543 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:19.174224 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.174249 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.175062 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.175069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.176494 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.176506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.177070 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.177075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.177251 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.177868 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:19.177896 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:19.179988 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:19.182510 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:19.182516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:19.182530 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:19.182828 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:19.183265 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:19.183268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:19.183269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:19.183429 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:19.184046 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:19.184053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:19.184406 (+ 353us) client_negotiation.cc:769] Sending connection context
0805 23:09:19.184601 (+ 195us) client_negotiation.cc:240] Negotiation successful
0805 23:09:19.184980 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":110,"threads_started":1}
I20260805 23:09:19.187443 31569 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40251
I20260805 23:09:19.187739 31569 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.67@KRBTEST.COM" created.
I20260805 23:09:19.188426 31569 heartbeater.cc:511] Master 127.3.12.126:40251 requested a full tablet report, sending...
I20260805 23:09:19.189648 31100 ts_manager.cc:194] Registered new tserver with Master: ba13052606364fa783e04a7c4aa5dd74 (127.3.12.66:43303)
I20260805 23:09:19.190253 31100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.66@KRBTEST.COM'} at 127.3.12.66:55393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
Entry for principal kudu/127.3.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab.
I20260805 23:09:19.208107 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40251
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:19.327477 31576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:19.327735 31576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:19.327780 31576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:19.331940 31576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:19.332012 31576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:19.332113 31576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:09:19.337479 31576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.67.keytab
--principal=kudu/127.3.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:40251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:19.338553 31576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:19.339613 31576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:19.350198 31583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:19.350224 31584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:19.350446 31576 server_base.cc:1034] running on GCE node
W20260805 23:09:19.350198 31588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:19.351030 31576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:19.351691 31576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:19.352849 31576 hybrid_clock.cc:648] HybridClock initialized: now 1785971359352820 us; error 57 us; skew 500 ppm
Aug 05 23:09:19 dist-test-slave-2d19 krb5kdc[31037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:19.355808 31576 init.cc:377] Logged in from keytab as kudu/127.3.12.67@KRBTEST.COM (short username kudu)
I20260805 23:09:19.356891 31576 webserver.cc:460] Webserver started at http://127.3.12.67:39253/ using document root <none> and password file <none>
I20260805 23:09:19.357508 31576 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:19.357575 31576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:19.357831 31576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:19.360128 31576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "da62e7c39ef04fdd98ce8411753f1db7"
format_stamp: "Formatted at 2026-08-05 23:09:19 on dist-test-slave-2d19"
server_key: "2c962b93f70b1b917b1aa044d89793e1"
server_key_iv: "8a1ebac7326c5e1b4ed67e1dec7f73aa"
server_key_version: "encryptionkey@0"
I20260805 23:09:19.360792 31576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "da62e7c39ef04fdd98ce8411753f1db7"
format_stamp: "Formatted at 2026-08-05 23:09:19 on dist-test-slave-2d19"
server_key: "2c962b93f70b1b917b1aa044d89793e1"
server_key_iv: "8a1ebac7326c5e1b4ed67e1dec7f73aa"
server_key_version: "encryptionkey@0"
I20260805 23:09:19.364837 31576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 23:09:19.367439 31598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:19.368747 31576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:19.368896 31576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "da62e7c39ef04fdd98ce8411753f1db7"
format_stamp: "Formatted at 2026-08-05 23:09:19 on dist-test-slave-2d19"
server_key: "2c962b93f70b1b917b1aa044d89793e1"
server_key_iv: "8a1ebac7326c5e1b4ed67e1dec7f73aa"
server_key_version: "encryptionkey@0"
I20260805 23:09:19.369014 31576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:19.393105 31576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:19.396867 31576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:19.397104 31576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:19.397768 31576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:19.398649 31576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:19.398698 31576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:19.398769 31576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:19.398811 31576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:19.408058 31576 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:37825
I20260805 23:09:19.408084 31756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:37825 every 8 connection(s)
I20260805 23:09:19.409091 31576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:09:19.416348 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 31576
I20260805 23:09:19.416497 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:09:19.416812 3121 external_mini_cluster.cc:1428] Setting key 06bc01b9dd2131bb51308a6ef2bdb9cb
Aug 05 23:09:19 dist-test-slave-2d19 krb5kdc[31037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.67@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:19.423154 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.410923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:35437 (local address 127.3.12.126:40251)
0805 23:09:19.411078 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.411082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.412018 (+ 936us) server_negotiation.cc:407] Connection header received
0805 23:09:19.413443 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.413447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.413494 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.413586 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:19.414778 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.415309 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.416584 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.416807 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.416929 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.419841 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:19.419844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:19.419846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:19.419875 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:19.421296 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:19.421785 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:19.421788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:19.421789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:19.421865 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:19.422175 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:19.422177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:19.422178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:19.422351 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:19.422474 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.422907 (+ 433us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.423036 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260805 23:09:19.423980 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.411301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40251 (local address 127.3.12.67:35437)
0805 23:09:19.411847 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:19.411891 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:19.413240 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:19.413778 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:19.413784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:19.414146 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:19.414634 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.414648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.415424 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.415427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.416405 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.416415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.416973 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.416977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.417208 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.417923 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:19.417950 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:19.419693 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:19.421410 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:19.421415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:19.421428 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:19.421694 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:19.421946 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:19.421949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:19.421950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:19.422089 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:19.422443 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:19.422448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:19.422689 (+ 241us) client_negotiation.cc:769] Sending connection context
0805 23:09:19.422843 (+ 154us) client_negotiation.cc:240] Negotiation successful
0805 23:09:19.423093 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":168,"threads_started":1}
I20260805 23:09:19.425174 31757 heartbeater.cc:347] Connected to a master server at 127.3.12.126:40251
I20260805 23:09:19.425431 31757 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:19.425936 31757 heartbeater.cc:511] Master 127.3.12.126:40251 requested a full tablet report, sending...
I20260805 23:09:19.426957 31100 ts_manager.cc:194] Registered new tserver with Master: da62e7c39ef04fdd98ce8411753f1db7 (127.3.12.67:37825)
I20260805 23:09:19.427562 31100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.12.67@KRBTEST.COM'} at 127.3.12.67:35437
I20260805 23:09:19.430871 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:09:19.440080 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.432898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.3.12.126:40251)
0805 23:09:19.433024 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.433029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.433254 (+ 225us) server_negotiation.cc:407] Connection header received
0805 23:09:19.433363 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.433365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.433403 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.433470 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:19.434164 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.434654 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.435434 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.435649 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.435727 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.437091 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:19.437094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:19.437096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:19.437119 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:19.438636 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:19.439034 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:19.439037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:19.439038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:19.439114 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:19.439349 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:19.439351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:19.439352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:19.439548 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:19.439665 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.439865 (+ 200us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.439962 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:09:19.442941 31100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:09:19.446302 31100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:09:19.461158 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.455033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.3.12.66:43303)
0805 23:09:19.455409 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.455421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.456168 (+ 747us) server_negotiation.cc:407] Connection header received
0805 23:09:19.456288 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.456297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.456520 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.456692 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:19.456985 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.457882 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.459521 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.460425 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.460536 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.460752 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.460862 (+ 110us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.460959 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":146,"threads_started":1}
I20260805 23:09:19.461205 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.454913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:43303 (local address 127.0.0.1:35638)
0805 23:09:19.455808 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:19.455859 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:19.456007 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:19.456684 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:19.456690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:19.456713 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:19.456862 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.456869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.458026 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.458029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.459366 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.459378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.460556 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.460560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.460672 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.460715 (+ 43us) client_negotiation.cc:769] Sending connection context
0805 23:09:19.460830 (+ 115us) client_negotiation.cc:240] Negotiation successful
0805 23:09:19.461003 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":74,"threads_started":1}
I20260805 23:09:19.461692 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.454998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:37825 (local address 127.0.0.1:38064)
0805 23:09:19.455963 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:19.455976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:19.456048 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:19.457210 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:19.457213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:19.457231 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:19.457327 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.457334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.458839 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.458842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.459894 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.459903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.461046 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.461050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.461135 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.461158 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:09:19.461208 (+ 50us) client_negotiation.cc:240] Negotiation successful
0805 23:09:19.461502 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":93,"thread_start_us":97,"threads_started":1}
I20260805 23:09:19.461731 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.455665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42453 (local address 127.0.0.1:54056)
0805 23:09:19.456235 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:19.456247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:19.456341 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:19.457089 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:19.457092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:19.457111 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:19.457205 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.457210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.458840 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.458843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.459948 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.459959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.461310 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.461316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.461430 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.461451 (+ 21us) client_negotiation.cc:769] Sending connection context
0805 23:09:19.461520 (+ 69us) client_negotiation.cc:240] Negotiation successful
0805 23:09:19.461588 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":52,"thread_start_us":96,"threads_started":1}
I20260805 23:09:19.462284 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.455902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.3.12.67:37825)
0805 23:09:19.456783 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.456794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.456812 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:09:19.456877 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.456881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.457052 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.457488 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:19.457637 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.458682 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.460037 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.460916 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.461874 (+ 958us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.462010 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.462059 (+ 49us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.462143 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":155,"threads_started":1}
I20260805 23:09:19.462560 31781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.456376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.3.12.65:42453)
0805 23:09:19.456649 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.456666 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.456689 (+ 23us) server_negotiation.cc:407] Connection header received
0805 23:09:19.456768 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.456774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.456952 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.457104 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:19.457282 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.458431 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.460107 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.460960 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.461067 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.462194 (+ 1127us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.462237 (+ 43us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.462304 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260805 23:09:19.464672 31472 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc077a3862945978da4eb4271ee6677 (DEFAULT_TABLE table=test-table [id=ddb2fb67b6504926acba31ea9c95bced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:19.465685 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc077a3862945978da4eb4271ee6677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:19.465303 31286 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc077a3862945978da4eb4271ee6677 (DEFAULT_TABLE table=test-table [id=ddb2fb67b6504926acba31ea9c95bced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:19.465304 31664 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc077a3862945978da4eb4271ee6677 (DEFAULT_TABLE table=test-table [id=ddb2fb67b6504926acba31ea9c95bced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:19.466287 31664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc077a3862945978da4eb4271ee6677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:19.466287 31286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc077a3862945978da4eb4271ee6677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:19.472854 31783 tablet_bootstrap.cc:492] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Bootstrap starting.
I20260805 23:09:19.473992 31785 tablet_bootstrap.cc:492] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Bootstrap starting.
I20260805 23:09:19.474005 31784 tablet_bootstrap.cc:492] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac: Bootstrap starting.
I20260805 23:09:19.475649 31783 tablet_bootstrap.cc:654] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:19.476215 31785 tablet_bootstrap.cc:654] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:19.476215 31784 tablet_bootstrap.cc:654] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:19.476552 31783 log.cc:824] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:19.477154 31784 log.cc:824] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:19.477437 31785 log.cc:824] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:19.478845 31783 tablet_bootstrap.cc:492] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: No bootstrap required, opened a new log
I20260805 23:09:19.479082 31783 ts_tablet_manager.cc:1397] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260805 23:09:19.479215 31785 tablet_bootstrap.cc:492] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: No bootstrap required, opened a new log
I20260805 23:09:19.479212 31784 tablet_bootstrap.cc:492] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac: No bootstrap required, opened a new log
I20260805 23:09:19.479403 31784 ts_tablet_manager.cc:1397] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:09:19.479403 31785 ts_tablet_manager.cc:1397] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260805 23:09:19.482123 31783 raft_consensus.cc:348] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.482388 31783 raft_consensus.cc:374] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:19.482445 31783 raft_consensus.cc:729] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba13052606364fa783e04a7c4aa5dd74, State: Initialized, Role: FOLLOWER
I20260805 23:09:19.482949 31783 consensus_queue.cc:260] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.483412 31785 raft_consensus.cc:348] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.483660 31569 heartbeater.cc:503] Master 127.3.12.126:40251 was elected leader, sending a full tablet report...
I20260805 23:09:19.483705 31785 raft_consensus.cc:374] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:19.483760 31785 raft_consensus.cc:729] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da62e7c39ef04fdd98ce8411753f1db7, State: Initialized, Role: FOLLOWER
I20260805 23:09:19.484074 31783 ts_tablet_manager.cc:1428] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260805 23:09:19.484184 31785 consensus_queue.cc:260] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.485183 31785 ts_tablet_manager.cc:1428] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 23:09:19.485502 31757 heartbeater.cc:503] Master 127.3.12.126:40251 was elected leader, sending a full tablet report...
I20260805 23:09:19.484906 31784 raft_consensus.cc:348] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.486886 31784 raft_consensus.cc:374] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:19.486961 31784 raft_consensus.cc:729] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa2ee279c6b48a69261522a0b45b1ac, State: Initialized, Role: FOLLOWER
I20260805 23:09:19.487391 31784 consensus_queue.cc:260] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.488407 31379 heartbeater.cc:503] Master 127.3.12.126:40251 was elected leader, sending a full tablet report...
I20260805 23:09:19.488444 31784 ts_tablet_manager.cc:1428] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac: Time spent starting tablet: real 0.009s user 0.001s sys 0.005s
W20260805 23:09:19.660116 31758 tablet.cc:2369] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260805 23:09:19.669036 31570 tablet.cc:2369] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:09:19.690541 31794 raft_consensus.cc:482] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:09:19.690857 31794 raft_consensus.cc:504] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.692193 31794 leader_election.cc:288] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba13052606364fa783e04a7c4aa5dd74 (127.3.12.66:43303), da62e7c39ef04fdd98ce8411753f1db7 (127.3.12.67:37825)
W20260805 23:09:19.694677 31380 tablet.cc:2369] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:09:19.698072 31780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.692714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:53935 (local address 127.3.12.67:37825)
0805 23:09:19.692857 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.692861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.693074 (+ 213us) server_negotiation.cc:407] Connection header received
0805 23:09:19.693139 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.693142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.693189 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.693281 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:19.693902 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.694702 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.696659 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.697286 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.697419 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.697488 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.697862 (+ 374us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.697941 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260805 23:09:19.698094 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.692539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:53855 (local address 127.3.12.66:43303)
0805 23:09:19.692775 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.692780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.692872 (+ 92us) server_negotiation.cc:407] Connection header received
0805 23:09:19.693127 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.693131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.693189 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.693289 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:19.693901 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.694707 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.696659 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.697316 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.697419 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.697488 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.697862 (+ 374us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.697990 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260805 23:09:19.698155 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.692403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:43303 (local address 127.3.12.65:53855)
0805 23:09:19.692778 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:19.692794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:19.692950 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:19.693312 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:19.693315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:19.693355 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:19.693762 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.693770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.694853 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.694857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.696504 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.696518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.697443 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.697447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.697689 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.697733 (+ 44us) client_negotiation.cc:769] Sending connection context
0805 23:09:19.697818 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 23:09:19.697960 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":360,"spinlock_wait_cycles":30208,"thread_start_us":135,"threads_started":1}
I20260805 23:09:19.698159 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.692643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:37825 (local address 127.3.12.65:53935)
0805 23:09:19.692951 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:19.692964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:19.693054 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:19.693312 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:19.693315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:19.693357 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:19.693764 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.693770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.694853 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.694857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.696504 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:19.696518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.697443 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.697447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:19.697662 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.697733 (+ 71us) client_negotiation.cc:769] Sending connection context
0805 23:09:19.697818 (+ 85us) client_negotiation.cc:240] Negotiation successful
0805 23:09:19.697968 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":433,"spinlock_wait_cycles":8448,"thread_start_us":113,"threads_started":1}
I20260805 23:09:19.698923 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc077a3862945978da4eb4271ee6677" candidate_uuid: "afa2ee279c6b48a69261522a0b45b1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba13052606364fa783e04a7c4aa5dd74" is_pre_election: true
I20260805 23:09:19.698923 31693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc077a3862945978da4eb4271ee6677" candidate_uuid: "afa2ee279c6b48a69261522a0b45b1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da62e7c39ef04fdd98ce8411753f1db7" is_pre_election: true
I20260805 23:09:19.699258 31693 raft_consensus.cc:2454] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afa2ee279c6b48a69261522a0b45b1ac in term 0.
I20260805 23:09:19.699271 31501 raft_consensus.cc:2454] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afa2ee279c6b48a69261522a0b45b1ac in term 0.
I20260805 23:09:19.699816 31222 leader_election.cc:302] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: afa2ee279c6b48a69261522a0b45b1ac, ba13052606364fa783e04a7c4aa5dd74, da62e7c39ef04fdd98ce8411753f1db7; no voters:
I20260805 23:09:19.700132 31794 raft_consensus.cc:2781] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:09:19.700234 31794 raft_consensus.cc:482] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:09:19.700322 31794 raft_consensus.cc:3037] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:19.701398 31794 raft_consensus.cc:504] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.701822 31794 leader_election.cc:288] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [CANDIDATE]: Term 1 election: Requested vote from peers ba13052606364fa783e04a7c4aa5dd74 (127.3.12.66:43303), da62e7c39ef04fdd98ce8411753f1db7 (127.3.12.67:37825)
I20260805 23:09:19.702157 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc077a3862945978da4eb4271ee6677" candidate_uuid: "afa2ee279c6b48a69261522a0b45b1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba13052606364fa783e04a7c4aa5dd74"
I20260805 23:09:19.702302 31501 raft_consensus.cc:3037] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:19.702324 31693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc077a3862945978da4eb4271ee6677" candidate_uuid: "afa2ee279c6b48a69261522a0b45b1ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da62e7c39ef04fdd98ce8411753f1db7"
I20260805 23:09:19.702464 31693 raft_consensus.cc:3037] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:19.703430 31501 raft_consensus.cc:2454] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afa2ee279c6b48a69261522a0b45b1ac in term 1.
I20260805 23:09:19.703523 31693 raft_consensus.cc:2454] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afa2ee279c6b48a69261522a0b45b1ac in term 1.
I20260805 23:09:19.703848 31223 leader_election.cc:302] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afa2ee279c6b48a69261522a0b45b1ac, ba13052606364fa783e04a7c4aa5dd74; no voters:
I20260805 23:09:19.704075 31794 raft_consensus.cc:2781] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:19.704397 31794 raft_consensus.cc:686] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 1 LEADER]: Becoming Leader. State: Replica: afa2ee279c6b48a69261522a0b45b1ac, State: Running, Role: LEADER
I20260805 23:09:19.704768 31794 consensus_queue.cc:237] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } }
I20260805 23:09:19.707899 31096 catalog_manager.cc:5495] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac reported cstate change: term changed from 0 to 1, leader changed from <none> to afa2ee279c6b48a69261522a0b45b1ac (127.3.12.65). New cstate: current_term: 1 leader_uuid: "afa2ee279c6b48a69261522a0b45b1ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afa2ee279c6b48a69261522a0b45b1ac" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 42453 } health_report { overall_health: HEALTHY } } }
I20260805 23:09:19.755957 31781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.752688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.3.12.65:42453)
0805 23:09:19.752826 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:19.752829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:19.752844 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:09:19.752925 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:19.752928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:19.752981 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:19.753085 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:19.753299 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.753795 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.754617 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:19.754824 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:19.754909 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:19.755200 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:19.755606 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:19.755698 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:19.755796 (+ 98us) server_negotiation.cc:299] Negotiation successful
0805 23:09:19.755845 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260805 23:09:19.762766 31501 raft_consensus.cc:1261] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 1 FOLLOWER]: Refusing update from remote peer afa2ee279c6b48a69261522a0b45b1ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:19.762774 31693 raft_consensus.cc:1261] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 1 FOLLOWER]: Refusing update from remote peer afa2ee279c6b48a69261522a0b45b1ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:19.763581 31794 consensus_queue.cc:1035] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba13052606364fa783e04a7c4aa5dd74" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 43303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:19.763804 31804 consensus_queue.cc:1035] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "da62e7c39ef04fdd98ce8411753f1db7" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 37825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:19.771389 31810 mvcc.cc:205] Tried to move back new op lower bound from 7315338689579409408 to 7315338689353216000. Current Snapshot: MvccSnapshot[applied={T|T < 7315338689579409408}]
I20260805 23:09:19.781337 31096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:19.781548 31096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:19.784451 31096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0f7a027cdd64dbaa75bfabcdb0fc0ae: Sending DeleteTablet for 3 replicas of tablet 2bc077a3862945978da4eb4271ee6677
I20260805 23:09:19.785319 31286 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc077a3862945978da4eb4271ee6677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:19 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:54056
I20260805 23:09:19.785367 31472 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc077a3862945978da4eb4271ee6677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:19 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:35638
I20260805 23:09:19.785953 31824 tablet_replica.cc:330] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac: stopping tablet replica
I20260805 23:09:19.785928 31664 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc077a3862945978da4eb4271ee6677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:19 UTC) from {username='kudu', principal='kudu/127.3.12.126@KRBTEST.COM'} at 127.0.0.1:38064
I20260805 23:09:19.786389 31824 raft_consensus.cc:2229] T 2bc077a3862945978da4eb4271ee6677 P afa2ee279c6b48a69261522a0b45b1ac [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:09:19.786420 31825 tablet_replica.cc:330] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: stopping tablet replica
I20260805 23:09:19.786743 31825 raft_consensus.cc:2229] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:19.786801 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 31192
I20260805 23:09:19.787122 31825 raft_consensus.cc:2258] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:19.789258 31825 ts_tablet_manager.cc:1905] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:19.792548 31826 tablet_replica.cc:330] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: stopping tablet replica
I20260805 23:09:19.793053 31826 raft_consensus.cc:2229] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:19.793274 31825 ts_tablet_manager.cc:1918] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:19.793416 31825 log.cc:1196] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/2bc077a3862945978da4eb4271ee6677
I20260805 23:09:19.793485 31826 raft_consensus.cc:2258] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:19.793853 31825 ts_tablet_manager.cc:1939] T 2bc077a3862945978da4eb4271ee6677 P ba13052606364fa783e04a7c4aa5dd74: Deleting consensus metadata
I20260805 23:09:19.795189 31826 ts_tablet_manager.cc:1905] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:19.795413 31081 catalog_manager.cc:4841] TS ba13052606364fa783e04a7c4aa5dd74 (127.3.12.66:43303): tablet 2bc077a3862945978da4eb4271ee6677 (table test-table [id=ddb2fb67b6504926acba31ea9c95bced]) successfully deleted
I20260805 23:09:19.798971 31826 ts_tablet_manager.cc:1918] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:19.799095 31826 log.cc:1196] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/2bc077a3862945978da4eb4271ee6677
I20260805 23:09:19.799481 31826 ts_tablet_manager.cc:1939] T 2bc077a3862945978da4eb4271ee6677 P da62e7c39ef04fdd98ce8411753f1db7: Deleting consensus metadata
W20260805 23:09:19.800819 31083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:42453 (error 108)
I20260805 23:09:19.800930 31080 catalog_manager.cc:4841] TS da62e7c39ef04fdd98ce8411753f1db7 (127.3.12.67:37825): tablet 2bc077a3862945978da4eb4271ee6677 (table test-table [id=ddb2fb67b6504926acba31ea9c95bced]) successfully deleted
I20260805 23:09:19.801841 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 31387
I20260805 23:09:19.801992 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:19.801606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:42453 (local address 127.0.0.1:54076)
0805 23:09:19.801810 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:19.801891 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260805 23:09:19.808907 31083 catalog_manager.cc:4584] TS afa2ee279c6b48a69261522a0b45b1ac (127.3.12.65:42453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2bc077a3862945978da4eb4271ee6677: Network error: Client connection negotiation failed: client connection to 127.3.12.65:42453: connect: Connection refused (error 111)
I20260805 23:09:19.815099 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 31576
I20260805 23:09:19.827919 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 31057
2026-08-05T23:09:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2753 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-05T23:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:19Z Disabled control of system clock
I20260805 23:09:19.869726 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:39487
--webserver_interface=127.3.12.126
--webserver_port=0
--builtin_ntp_servers=127.3.12.84:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:39487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 23:09:20.004837 31833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:20.005138 31833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:20.005177 31833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:20.009271 31833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:20.009338 31833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:20.009354 31833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:20.009410 31833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:20.009442 31833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:20.014266 31833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:39487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:20.015415 31833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:20.016866 31833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:20.025610 31840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:20.025610 31841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:20.025842 31833 server_base.cc:1034] running on GCE node
W20260805 23:09:20.025605 31844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:20.026669 31833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:20.027801 31833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:20.028967 31833 hybrid_clock.cc:648] HybridClock initialized: now 1785971360028940 us; error 66 us; skew 500 ppm
I20260805 23:09:20.030898 31833 webserver.cc:460] Webserver started at http://127.3.12.126:37475/ using document root <none> and password file <none>
I20260805 23:09:20.031515 31833 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:20.031581 31833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:20.031814 31833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:20.034031 31833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "4815a52dc4d047b88084f96c37189528"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "dcdfa49996b57a31b0aacdde63b00ad8"
server_key_iv: "bd64b0ac1d8cdba510f202f7a36ac176"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.034646 31833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "4815a52dc4d047b88084f96c37189528"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "dcdfa49996b57a31b0aacdde63b00ad8"
server_key_iv: "bd64b0ac1d8cdba510f202f7a36ac176"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.038558 31833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260805 23:09:20.041031 31852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.042294 31833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:20.042424 31833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "4815a52dc4d047b88084f96c37189528"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "dcdfa49996b57a31b0aacdde63b00ad8"
server_key_iv: "bd64b0ac1d8cdba510f202f7a36ac176"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.042533 31833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:20.065032 31833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:20.065826 31833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:20.066068 31833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:20.073589 31833 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:39487
I20260805 23:09:20.073601 31924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:39487 every 8 connection(s)
I20260805 23:09:20.074591 31833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:20.076695 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 31833
I20260805 23:09:20.076836 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:20.077224 3121 external_mini_cluster.cc:1428] Setting key f6f58eb3bc9f501b9a80e7f4499a20f2
I20260805 23:09:20.078455 31925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:20.085945 31925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528: Bootstrap starting.
I20260805 23:09:20.087667 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.079013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.3.12.126:39487)
0805 23:09:20.079729 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.079750 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.079786 (+ 36us) server_negotiation.cc:407] Connection header received
0805 23:09:20.080643 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.080668 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.081181 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.081555 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:20.082072 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.083028 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.084244 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.084788 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.084908 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.085841 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:20.085846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:20.085858 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:20.086321 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:20.086379 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:20.086400 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:20.086476 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.086657 (+ 181us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.086962 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":156,"threads_started":1}
I20260805 23:09:20.088451 31925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:20.089214 31925 log.cc:824] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:20.091116 31925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528: No bootstrap required, opened a new log
I20260805 23:09:20.094619 31925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4815a52dc4d047b88084f96c37189528" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39487 } }
I20260805 23:09:20.094882 31925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:20.094971 31925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4815a52dc4d047b88084f96c37189528, State: Initialized, Role: FOLLOWER
I20260805 23:09:20.095525 31925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4815a52dc4d047b88084f96c37189528" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39487 } }
I20260805 23:09:20.095693 31925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:20.095765 31925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:20.095867 31925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:20.096892 31925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4815a52dc4d047b88084f96c37189528" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39487 } }
I20260805 23:09:20.097241 31925 leader_election.cc:302] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4815a52dc4d047b88084f96c37189528; no voters:
I20260805 23:09:20.097565 31925 leader_election.cc:288] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:20.097719 31935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:20.097947 31935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [term 1 LEADER]: Becoming Leader. State: Replica: 4815a52dc4d047b88084f96c37189528, State: Running, Role: LEADER
I20260805 23:09:20.098258 31935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4815a52dc4d047b88084f96c37189528" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39487 } }
I20260805 23:09:20.098951 31925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:20.100179 31937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4815a52dc4d047b88084f96c37189528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4815a52dc4d047b88084f96c37189528" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39487 } } }
I20260805 23:09:20.100179 31938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4815a52dc4d047b88084f96c37189528. Latest consensus state: current_term: 1 leader_uuid: "4815a52dc4d047b88084f96c37189528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4815a52dc4d047b88084f96c37189528" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39487 } } }
I20260805 23:09:20.100412 31937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:20.100410 31938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:20.100919 31949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:20.103514 31949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:20.109282 31949 catalog_manager.cc:1347] Generated new cluster ID: ecbac8a4e55745e4980e2b060043efdd
I20260805 23:09:20.109349 31949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:20.114284 31949 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:20.115144 31949 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:20.123015 31949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528: Generated new TSK 0
I20260805 23:09:20.123791 31949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:20.134434 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.65:0
--local_ip_for_outbound_sockets=127.3.12.65
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39487
--builtin_ntp_servers=127.3.12.84:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260805 23:09:20.248693 31966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:20.248989 31966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:20.249073 31966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:20.253114 31966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:20.253190 31966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:20.253278 31966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.65
I20260805 23:09:20.257943 31966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.12.65
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.65
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:20.259054 31966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:20.260599 31966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:20.270097 31975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:20.270200 31966 server_base.cc:1034] running on GCE node
W20260805 23:09:20.270109 31976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:20.270146 31979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:20.270701 31966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:20.271414 31966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:20.272583 31966 hybrid_clock.cc:648] HybridClock initialized: now 1785971360272565 us; error 25 us; skew 500 ppm
I20260805 23:09:20.274413 31966 webserver.cc:460] Webserver started at http://127.3.12.65:45571/ using document root <none> and password file <none>
I20260805 23:09:20.275070 31966 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:20.275170 31966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:20.275416 31966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:20.277508 31966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/data/instance:
uuid: "75a5136b00c34bc2a072f324fad4c314"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "f92b008db6ae06f6fa2f9475d4478f06"
server_key_iv: "0f2fbbb66b73d33fd4f6c13a4daac234"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.278093 31966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance:
uuid: "75a5136b00c34bc2a072f324fad4c314"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "f92b008db6ae06f6fa2f9475d4478f06"
server_key_iv: "0f2fbbb66b73d33fd4f6c13a4daac234"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.281810 31966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260805 23:09:20.284372 31986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.285619 31966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:09:20.285754 31966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/wal
uuid: "75a5136b00c34bc2a072f324fad4c314"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "f92b008db6ae06f6fa2f9475d4478f06"
server_key_iv: "0f2fbbb66b73d33fd4f6c13a4daac234"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.285873 31966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:20.302110 31966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:20.302851 31966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:20.303074 31966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:20.303709 31966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:20.304790 31966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:20.304836 31966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.304904 31966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:20.304939 31966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.314216 31966 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.65:40183
I20260805 23:09:20.314244 32147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.65:40183 every 8 connection(s)
I20260805 23:09:20.315124 31966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/data/info.pb
I20260805 23:09:20.322116 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 31966
I20260805 23:09:20.322256 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-0/wal/instance
I20260805 23:09:20.322584 3121 external_mini_cluster.cc:1428] Setting key d3012aa79c842cdcd005be5ffe6da52c
I20260805 23:09:20.324950 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.66:0
--local_ip_for_outbound_sockets=127.3.12.66
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39487
--builtin_ntp_servers=127.3.12.84:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 23:09:20.325495 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.317099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.65:59569 (local address 127.3.12.126:39487)
0805 23:09:20.317283 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.317288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.318409 (+ 1121us) server_negotiation.cc:407] Connection header received
0805 23:09:20.319807 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.319810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.319860 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.319935 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:20.320911 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.321455 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.323155 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.323452 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.323559 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.324869 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:20.324872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:20.324874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:20.324886 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:20.324905 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:20.324913 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:20.324985 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.325216 (+ 231us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.325300 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260805 23:09:20.327100 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.317462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:39487 (local address 127.3.12.65:59569)
0805 23:09:20.318250 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.318292 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:20.319614 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:20.320101 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:20.320108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:20.320190 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:20.320762 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.320776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.321584 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.321587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.323006 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.323017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.323607 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.323610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.323754 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.324547 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:20.324549 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:20.324600 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:20.324685 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:20.325018 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:20.325030 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:20.325038 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 23:09:20.325677 (+ 639us) client_negotiation.cc:240] Negotiation successful
0805 23:09:20.326040 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":113,"threads_started":1}
I20260805 23:09:20.328521 32148 heartbeater.cc:347] Connected to a master server at 127.3.12.126:39487
I20260805 23:09:20.328774 32148 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:20.329411 32148 heartbeater.cc:511] Master 127.3.12.126:39487 requested a full tablet report, sending...
I20260805 23:09:20.331032 31878 ts_manager.cc:194] Registered new tserver with Master: 75a5136b00c34bc2a072f324fad4c314 (127.3.12.65:40183)
I20260805 23:09:20.332204 31878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.65:59569
W20260805 23:09:20.444060 32151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:20.444353 32151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:20.444393 32151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:20.448477 32151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:20.448539 32151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:20.448630 32151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.66
I20260805 23:09:20.453649 32151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.12.66
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.66
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:20.454790 32151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:20.456231 32151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:20.466107 32167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:20.466101 32161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:20.466101 32165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:20.466744 32151 server_base.cc:1034] running on GCE node
I20260805 23:09:20.467116 32151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:20.467690 32151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:20.468851 32151 hybrid_clock.cc:648] HybridClock initialized: now 1785971360468834 us; error 29 us; skew 500 ppm
I20260805 23:09:20.470825 32151 webserver.cc:460] Webserver started at http://127.3.12.66:37753/ using document root <none> and password file <none>
I20260805 23:09:20.471419 32151 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:20.471482 32151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:20.471704 32151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:20.473887 32151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/data/instance:
uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "4d4bc87fc5da460ad62fb3ef76d51bc0"
server_key_iv: "ce1427a578e933c78a602a96c0ad870c"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.474498 32151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance:
uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "4d4bc87fc5da460ad62fb3ef76d51bc0"
server_key_iv: "ce1427a578e933c78a602a96c0ad870c"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.478466 32151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:20.481058 32175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.482316 32151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:20.482460 32151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/wal
uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "4d4bc87fc5da460ad62fb3ef76d51bc0"
server_key_iv: "ce1427a578e933c78a602a96c0ad870c"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.482594 32151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:20.493005 32151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:20.493742 32151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:20.493960 32151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:20.494588 32151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:20.495639 32151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:20.495718 32151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.495795 32151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:20.495874 32151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.505311 32151 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.66:34561
I20260805 23:09:20.505342 32331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.66:34561 every 8 connection(s)
I20260805 23:09:20.506253 32151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/data/info.pb
I20260805 23:09:20.513787 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 32151
I20260805 23:09:20.514086 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/wal/instance
I20260805 23:09:20.514401 3121 external_mini_cluster.cc:1428] Setting key 6761e255eff06c20fc0599c55cff31ea
I20260805 23:09:20.516424 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.508352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:37711 (local address 127.3.12.126:39487)
0805 23:09:20.508503 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.508507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.509640 (+ 1133us) server_negotiation.cc:407] Connection header received
0805 23:09:20.511078 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.511081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.511133 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.511219 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:20.512189 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.512749 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.513976 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.514328 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.514456 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.515772 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:20.515775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:20.515778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:20.515791 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:20.515843 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:20.515855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:20.515931 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.516184 (+ 253us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.516278 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260805 23:09:20.516877 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.12.67:0
--local_ip_for_outbound_sockets=127.3.12.67
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39487
--builtin_ntp_servers=127.3.12.84:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260805 23:09:20.517416 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.508804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:39487 (local address 127.3.12.66:37711)
0805 23:09:20.509471 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.509513 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:20.510795 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:20.511398 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:20.511405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:20.511490 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:20.511997 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.512011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.512904 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.512907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.513823 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.513833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.514501 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.514505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.514680 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.515466 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:20.515468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:20.515522 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:20.515603 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:20.515961 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:20.515974 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:20.515982 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 23:09:20.516150 (+ 168us) client_negotiation.cc:240] Negotiation successful
0805 23:09:20.516468 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260805 23:09:20.518683 32332 heartbeater.cc:347] Connected to a master server at 127.3.12.126:39487
I20260805 23:09:20.518986 32332 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:20.519562 32332 heartbeater.cc:511] Master 127.3.12.126:39487 requested a full tablet report, sending...
I20260805 23:09:20.520744 31878 ts_manager.cc:194] Registered new tserver with Master: 6795acd79efc40a4aef3f5ea3cbf1d2b (127.3.12.66:34561)
I20260805 23:09:20.521606 31878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.66:37711
W20260805 23:09:20.634037 32335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:20.634325 32335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:20.634361 32335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:20.638394 32335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:20.638449 32335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:20.638504 32335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.12.67
I20260805 23:09:20.643275 32335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.12.67
--webserver_port=0
--tserver_master_addrs=127.3.12.126:39487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.12.67
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:20.644464 32335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:20.645932 32335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:20.656551 32343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:20.656701 32349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:20.656592 32335 server_base.cc:1034] running on GCE node
W20260805 23:09:20.656599 32344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:20.657418 32335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:20.658300 32335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:20.659525 32335 hybrid_clock.cc:648] HybridClock initialized: now 1785971360659504 us; error 43 us; skew 500 ppm
I20260805 23:09:20.661664 32335 webserver.cc:460] Webserver started at http://127.3.12.67:35151/ using document root <none> and password file <none>
I20260805 23:09:20.662308 32335 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:20.662513 32335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:20.662793 32335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:20.665007 32335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/data/instance:
uuid: "d66d1c5cc2c046988adc912fc9d0ffcd"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "706466037eeb4fc9386e0771c0371fc9"
server_key_iv: "3565a466d5e3054b33063e98e043de70"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.665603 32335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance:
uuid: "d66d1c5cc2c046988adc912fc9d0ffcd"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "706466037eeb4fc9386e0771c0371fc9"
server_key_iv: "3565a466d5e3054b33063e98e043de70"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.669445 32335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260805 23:09:20.671998 32356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.673358 32335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:09:20.673509 32335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/wal
uuid: "d66d1c5cc2c046988adc912fc9d0ffcd"
format_stamp: "Formatted at 2026-08-05 23:09:20 on dist-test-slave-2d19"
server_key: "706466037eeb4fc9386e0771c0371fc9"
server_key_iv: "3565a466d5e3054b33063e98e043de70"
server_key_version: "encryptionkey@0"
I20260805 23:09:20.673627 32335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:20.685005 32335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:20.685770 32335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:20.685958 32335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:20.686529 32335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 23:09:20.687533 32335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 23:09:20.687608 32335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.687674 32335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 23:09:20.687731 32335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:20.697412 32335 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.67:43995
I20260805 23:09:20.697505 32528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.67:43995 every 8 connection(s)
I20260805 23:09:20.698415 32335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/data/info.pb
I20260805 23:09:20.705765 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 32335
I20260805 23:09:20.705905 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/wal/instance
I20260805 23:09:20.706193 3121 external_mini_cluster.cc:1428] Setting key 5a4e4c2954c165e312442d5bea1d35e3
I20260805 23:09:20.708588 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.700581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.67:56763 (local address 127.3.12.126:39487)
0805 23:09:20.700731 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.700735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.701896 (+ 1161us) server_negotiation.cc:407] Connection header received
0805 23:09:20.703338 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.703341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.703389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.703467 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:20.704441 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.704981 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.706277 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.706578 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.706667 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.707991 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:20.707995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:20.708000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:20.708017 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:20.708042 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:20.708057 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:20.708141 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.708362 (+ 221us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.708426 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:09:20.709439 32533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.701053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:39487 (local address 127.3.12.67:56763)
0805 23:09:20.701763 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.701806 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:20.703125 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:20.703644 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:20.703651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:20.703735 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:20.704306 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.704322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.705129 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.705131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.706147 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.706157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.706755 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.706762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.706919 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.707714 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0805 23:09:20.707716 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:20.707758 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0805 23:09:20.707825 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:20.708156 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:20.708168 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:20.708176 (+ 8us) client_negotiation.cc:769] Sending connection context
0805 23:09:20.708353 (+ 177us) client_negotiation.cc:240] Negotiation successful
0805 23:09:20.708648 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":115,"threads_started":1}
I20260805 23:09:20.710606 32530 heartbeater.cc:347] Connected to a master server at 127.3.12.126:39487
I20260805 23:09:20.710861 32530 heartbeater.cc:464] Registering TS with master...
I20260805 23:09:20.711409 32530 heartbeater.cc:511] Master 127.3.12.126:39487 requested a full tablet report, sending...
I20260805 23:09:20.712538 31878 ts_manager.cc:194] Registered new tserver with Master: d66d1c5cc2c046988adc912fc9d0ffcd (127.3.12.67:43995)
I20260805 23:09:20.713130 31878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.12.67:56763
I20260805 23:09:20.720767 3121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260805 23:09:20.727046 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.722865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.3.12.126:39487)
0805 23:09:20.723069 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.723073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.723247 (+ 174us) server_negotiation.cc:407] Connection header received
0805 23:09:20.723428 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.723431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.723501 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.723601 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:20.723916 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.724516 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.725397 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.725621 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.725738 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.726583 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:20.726587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:20.726589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:20.726601 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:20.726619 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:20.726626 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:20.726709 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.726836 (+ 127us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.726892 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260805 23:09:20.730165 31878 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 23:09:20.733349 31878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 23:09:20.748301 32553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.741814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.66:34561 (local address 127.0.0.1:40886)
0805 23:09:20.742571 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.742626 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:20.742771 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:20.743620 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:20.743626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:20.743653 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:20.743817 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.743824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.745116 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.745121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.746600 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.746613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.747693 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.747696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.747814 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.747852 (+ 38us) client_negotiation.cc:769] Sending connection context
0805 23:09:20.747973 (+ 121us) client_negotiation.cc:240] Negotiation successful
0805 23:09:20.748101 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":49,"spinlock_wait_cycles":10368,"thread_start_us":77,"threads_started":1}
I20260805 23:09:20.748453 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.741925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:40183 (local address 127.0.0.1:48946)
0805 23:09:20.742571 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.742626 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:20.742771 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:20.743815 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:20.743820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:20.743838 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:20.743961 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.743968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.745625 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.745631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.746938 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.746949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.747959 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.747962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.748047 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.748070 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:09:20.748135 (+ 65us) client_negotiation.cc:240] Negotiation successful
0805 23:09:20.748209 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":128,"spinlock_wait_cycles":3840,"thread_start_us":97,"threads_started":1}
I20260805 23:09:20.748456 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.742658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.3.12.65:40183)
0805 23:09:20.743272 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.743285 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.743310 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:09:20.743390 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.743395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.743602 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.744107 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:20.744304 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.745420 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.747085 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.747851 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.747954 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.748140 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.748191 (+ 51us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.748308 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":127,"threads_started":1}
I20260805 23:09:20.749078 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.743023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.3.12.67:43995)
0805 23:09:20.743469 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.743479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.743500 (+ 21us) server_negotiation.cc:407] Connection header received
0805 23:09:20.743621 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.743625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.743808 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.743960 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:20.744198 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.745420 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.747072 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.747698 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.747806 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.747998 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.748832 (+ 834us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.748916 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":146,"threads_started":1}
I20260805 23:09:20.749084 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.741981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.3.12.66:34561)
0805 23:09:20.743060 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.743068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.743121 (+ 53us) server_negotiation.cc:407] Connection header received
0805 23:09:20.743202 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.743211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.743434 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.743998 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:20.744136 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.744995 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.746719 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.747534 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.747661 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.747865 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.748797 (+ 932us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.748902 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":181,"threads_started":1}
I20260805 23:09:20.749473 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.742193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43995 (local address 127.0.0.1:46970)
0805 23:09:20.742724 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.742737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:20.742806 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:20.743955 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:20.743958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:20.743977 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:20.744068 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.744074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.745601 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.745606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.746939 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.746947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.748058 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.748071 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.748145 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.748166 (+ 21us) client_negotiation.cc:769] Sending connection context
0805 23:09:20.748236 (+ 70us) client_negotiation.cc:240] Negotiation successful
0805 23:09:20.748344 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":114,"thread_start_us":90,"threads_started":1}
I20260805 23:09:20.751714 32243 tablet_service.cc:1467] Processing CreateTablet for tablet b7ccb18bc623430aacdf7d036e4ba7b8 (DEFAULT_TABLE table=test-table [id=65a366f5721b4cac9ff082764e278079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:20.751717 32058 tablet_service.cc:1467] Processing CreateTablet for tablet b7ccb18bc623430aacdf7d036e4ba7b8 (DEFAULT_TABLE table=test-table [id=65a366f5721b4cac9ff082764e278079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:20.752954 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ccb18bc623430aacdf7d036e4ba7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:20.752954 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ccb18bc623430aacdf7d036e4ba7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:20.754738 32434 tablet_service.cc:1467] Processing CreateTablet for tablet b7ccb18bc623430aacdf7d036e4ba7b8 (DEFAULT_TABLE table=test-table [id=65a366f5721b4cac9ff082764e278079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260805 23:09:20.755631 32434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ccb18bc623430aacdf7d036e4ba7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:20.760553 32560 tablet_bootstrap.cc:492] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: Bootstrap starting.
I20260805 23:09:20.761018 32558 tablet_bootstrap.cc:492] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314: Bootstrap starting.
I20260805 23:09:20.761988 32562 tablet_bootstrap.cc:492] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: Bootstrap starting.
I20260805 23:09:20.763500 32558 tablet_bootstrap.cc:654] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:20.764150 32562 tablet_bootstrap.cc:654] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:20.764410 32560 tablet_bootstrap.cc:654] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:20.764427 32558 log.cc:824] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:20.765185 32560 log.cc:824] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:20.765168 32562 log.cc:824] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:20.766624 32558 tablet_bootstrap.cc:492] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314: No bootstrap required, opened a new log
I20260805 23:09:20.766849 32558 ts_tablet_manager.cc:1397] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260805 23:09:20.767120 32562 tablet_bootstrap.cc:492] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: No bootstrap required, opened a new log
I20260805 23:09:20.767123 32560 tablet_bootstrap.cc:492] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: No bootstrap required, opened a new log
I20260805 23:09:20.767308 32562 ts_tablet_manager.cc:1397] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260805 23:09:20.767307 32560 ts_tablet_manager.cc:1397] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260805 23:09:20.770195 32558 raft_consensus.cc:348] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.770449 32558 raft_consensus.cc:374] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:20.770516 32558 raft_consensus.cc:729] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a5136b00c34bc2a072f324fad4c314, State: Initialized, Role: FOLLOWER
I20260805 23:09:20.771008 32558 consensus_queue.cc:260] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.771710 32148 heartbeater.cc:503] Master 127.3.12.126:39487 was elected leader, sending a full tablet report...
I20260805 23:09:20.771634 32562 raft_consensus.cc:348] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.771884 32562 raft_consensus.cc:374] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:20.771971 32562 raft_consensus.cc:729] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66d1c5cc2c046988adc912fc9d0ffcd, State: Initialized, Role: FOLLOWER
I20260805 23:09:20.772078 32558 ts_tablet_manager.cc:1428] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260805 23:09:20.772442 32562 consensus_queue.cc:260] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.773013 32560 raft_consensus.cc:348] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.773219 32530 heartbeater.cc:503] Master 127.3.12.126:39487 was elected leader, sending a full tablet report...
I20260805 23:09:20.773254 32560 raft_consensus.cc:374] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:20.773300 32560 raft_consensus.cc:729] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6795acd79efc40a4aef3f5ea3cbf1d2b, State: Initialized, Role: FOLLOWER
I20260805 23:09:20.773408 32562 ts_tablet_manager.cc:1428] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 23:09:20.773734 32560 consensus_queue.cc:260] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.774394 32332 heartbeater.cc:503] Master 127.3.12.126:39487 was elected leader, sending a full tablet report...
I20260805 23:09:20.774961 32560 ts_tablet_manager.cc:1428] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
W20260805 23:09:20.816496 32149 tablet.cc:2369] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 23:09:20.859364 32568 raft_consensus.cc:482] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 23:09:20.859645 32568 raft_consensus.cc:504] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.861006 32568 leader_election.cc:288] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75a5136b00c34bc2a072f324fad4c314 (127.3.12.65:40183), d66d1c5cc2c046988adc912fc9d0ffcd (127.3.12.67:43995)
I20260805 23:09:20.865379 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.861276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:40183 (local address 127.3.12.66:50747)
0805 23:09:20.861501 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.861518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:20.861620 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:20.862033 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:20.862036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:20.862080 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:20.862227 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.862235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.863155 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.863159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.864091 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.864101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.864885 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.864888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.865053 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.865081 (+ 28us) client_negotiation.cc:769] Sending connection context
0805 23:09:20.865172 (+ 91us) client_negotiation.cc:240] Negotiation successful
0805 23:09:20.865244 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1,"spinlock_wait_cycles":27392}
I20260805 23:09:20.865435 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.861545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:50747 (local address 127.3.12.65:40183)
0805 23:09:20.861698 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.861702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.861717 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:09:20.861776 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.861778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.861829 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.861950 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:20.862373 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.863030 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.864242 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.864756 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.864842 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.864895 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.865214 (+ 319us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.865292 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 23:09:20.865697 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.861276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.67:43995 (local address 127.3.12.66:55247)
0805 23:09:20.861903 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.861920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:20.862087 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:20.862488 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:20.862491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:20.862508 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0805 23:09:20.862628 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.862634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.863482 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.863485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.864424 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:20.864436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.865234 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.865238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:20.865333 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.865356 (+ 23us) client_negotiation.cc:769] Sending connection context
0805 23:09:20.865436 (+ 80us) client_negotiation.cc:240] Negotiation successful
0805 23:09:20.865533 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":307,"spinlock_wait_cycles":6400,"thread_start_us":156,"threads_started":1}
I20260805 23:09:20.865708 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.861373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.12.66:55247 (local address 127.3.12.67:43995)
0805 23:09:20.861513 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.861519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.861997 (+ 478us) server_negotiation.cc:407] Connection header received
0805 23:09:20.862217 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.862220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.862276 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.862330 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0805 23:09:20.862746 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.863328 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.864571 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.865086 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.865193 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.865249 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.865477 (+ 228us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.865560 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:09:20.866134 32086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ccb18bc623430aacdf7d036e4ba7b8" candidate_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a5136b00c34bc2a072f324fad4c314" is_pre_election: true
I20260805 23:09:20.866350 32461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ccb18bc623430aacdf7d036e4ba7b8" candidate_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" is_pre_election: true
I20260805 23:09:20.866469 32086 raft_consensus.cc:2454] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6795acd79efc40a4aef3f5ea3cbf1d2b in term 0.
I20260805 23:09:20.866670 32461 raft_consensus.cc:2454] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6795acd79efc40a4aef3f5ea3cbf1d2b in term 0.
I20260805 23:09:20.867043 32182 leader_election.cc:302] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6795acd79efc40a4aef3f5ea3cbf1d2b, 75a5136b00c34bc2a072f324fad4c314; no voters:
I20260805 23:09:20.867372 32568 raft_consensus.cc:2781] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 23:09:20.867466 32568 raft_consensus.cc:482] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 23:09:20.867518 32568 raft_consensus.cc:3037] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:20.868643 32568 raft_consensus.cc:504] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.869009 32568 leader_election.cc:288] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [CANDIDATE]: Term 1 election: Requested vote from peers 75a5136b00c34bc2a072f324fad4c314 (127.3.12.65:40183), d66d1c5cc2c046988adc912fc9d0ffcd (127.3.12.67:43995)
I20260805 23:09:20.869328 32086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ccb18bc623430aacdf7d036e4ba7b8" candidate_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75a5136b00c34bc2a072f324fad4c314"
I20260805 23:09:20.869441 32461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ccb18bc623430aacdf7d036e4ba7b8" candidate_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd"
I20260805 23:09:20.869462 32086 raft_consensus.cc:3037] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:20.869575 32461 raft_consensus.cc:3037] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:20.870574 32086 raft_consensus.cc:2454] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6795acd79efc40a4aef3f5ea3cbf1d2b in term 1.
I20260805 23:09:20.870730 32461 raft_consensus.cc:2454] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6795acd79efc40a4aef3f5ea3cbf1d2b in term 1.
I20260805 23:09:20.870936 32182 leader_election.cc:302] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6795acd79efc40a4aef3f5ea3cbf1d2b, 75a5136b00c34bc2a072f324fad4c314; no voters:
I20260805 23:09:20.871156 32568 raft_consensus.cc:2781] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:20.871425 32568 raft_consensus.cc:686] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 1 LEADER]: Becoming Leader. State: Replica: 6795acd79efc40a4aef3f5ea3cbf1d2b, State: Running, Role: LEADER
I20260805 23:09:20.871766 32568 consensus_queue.cc:237] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } }
I20260805 23:09:20.875480 31875 catalog_manager.cc:5495] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6795acd79efc40a4aef3f5ea3cbf1d2b (127.3.12.66). New cstate: current_term: 1 leader_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6795acd79efc40a4aef3f5ea3cbf1d2b" member_type: VOTER last_known_addr { host: "127.3.12.66" port: 34561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 } health_report { overall_health: UNKNOWN } } }
I20260805 23:09:20.900997 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.897378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.3.12.66:34561)
0805 23:09:20.897510 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:20.897513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:20.897528 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:09:20.897611 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:20.897614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:20.897671 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:20.897773 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:20.897991 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.898492 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.899446 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:20.899694 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:20.899776 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:20.900085 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:20.900603 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:20.900700 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:20.900832 (+ 132us) server_negotiation.cc:299] Negotiation successful
0805 23:09:20.900883 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:09:20.906972 32086 raft_consensus.cc:1261] T b7ccb18bc623430aacdf7d036e4ba7b8 P 75a5136b00c34bc2a072f324fad4c314 [term 1 FOLLOWER]: Refusing update from remote peer 6795acd79efc40a4aef3f5ea3cbf1d2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:20.906972 32461 raft_consensus.cc:1261] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 1 FOLLOWER]: Refusing update from remote peer 6795acd79efc40a4aef3f5ea3cbf1d2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:20.907714 32568 consensus_queue.cc:1035] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66d1c5cc2c046988adc912fc9d0ffcd" member_type: VOTER last_known_addr { host: "127.3.12.67" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:20.907917 32573 consensus_queue.cc:1035] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a5136b00c34bc2a072f324fad4c314" member_type: VOTER last_known_addr { host: "127.3.12.65" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:20.914970 32580 mvcc.cc:205] Tried to move back new op lower bound from 7315338694267584512 to 7315338694133846016. Current Snapshot: MvccSnapshot[applied={T|T < 7315338694267584512}]
I20260805 23:09:20.924181 31875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:20.924419 31875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260805 23:09:20.926810 31875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4815a52dc4d047b88084f96c37189528: Sending DeleteTablet for 3 replicas of tablet b7ccb18bc623430aacdf7d036e4ba7b8
I20260805 23:09:20.927582 32243 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ccb18bc623430aacdf7d036e4ba7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:20 UTC) from {username='slave'} at 127.0.0.1:40886
I20260805 23:09:20.927642 32058 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ccb18bc623430aacdf7d036e4ba7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:20 UTC) from {username='slave'} at 127.0.0.1:48946
I20260805 23:09:20.928227 32593 tablet_replica.cc:330] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: stopping tablet replica
I20260805 23:09:20.928231 32434 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ccb18bc623430aacdf7d036e4ba7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-05 23:09:20 UTC) from {username='slave'} at 127.0.0.1:46970
I20260805 23:09:20.928414 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 31966
I20260805 23:09:20.928630 32593 raft_consensus.cc:2229] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 1 LEADER]: Raft consensus shutting down.
I20260805 23:09:20.929044 32593 raft_consensus.cc:2258] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:20.935294 32593 ts_tablet_manager.cc:1905] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260805 23:09:20.939039 32593 ts_tablet_manager.cc:1918] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:20.939162 32593 log.cc:1196] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-1/wal/wals/b7ccb18bc623430aacdf7d036e4ba7b8
I20260805 23:09:20.939495 32593 ts_tablet_manager.cc:1939] T b7ccb18bc623430aacdf7d036e4ba7b8 P 6795acd79efc40a4aef3f5ea3cbf1d2b: Deleting consensus metadata
I20260805 23:09:20.940953 31855 catalog_manager.cc:4841] TS 6795acd79efc40a4aef3f5ea3cbf1d2b (127.3.12.66:34561): tablet b7ccb18bc623430aacdf7d036e4ba7b8 (table test-table [id=65a366f5721b4cac9ff082764e278079]) successfully deleted
I20260805 23:09:20.940986 32596 tablet_replica.cc:330] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: stopping tablet replica
I20260805 23:09:20.941375 32596 raft_consensus.cc:2229] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 23:09:20.941742 32596 raft_consensus.cc:2258] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 23:09:20.943382 32596 ts_tablet_manager.cc:1905] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260805 23:09:20.946487 31857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.12.65:40183 (error 108)
I20260805 23:09:20.946806 32596 ts_tablet_manager.cc:1918] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260805 23:09:20.946916 32596 log.cc:1196] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: Deleting WAL directory at /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785971095475883-3121-0/minicluster-data/ts-2/wal/wals/b7ccb18bc623430aacdf7d036e4ba7b8
I20260805 23:09:20.947279 32596 ts_tablet_manager.cc:1939] T b7ccb18bc623430aacdf7d036e4ba7b8 P d66d1c5cc2c046988adc912fc9d0ffcd: Deleting consensus metadata
I20260805 23:09:20.947777 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:20.947458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.65:40183 (local address 127.0.0.1:48948)
0805 23:09:20.947612 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:20.947688 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.65:40183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260805 23:09:20.947854 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 32151
W20260805 23:09:20.951323 31857 catalog_manager.cc:4584] TS 75a5136b00c34bc2a072f324fad4c314 (127.3.12.65:40183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7ccb18bc623430aacdf7d036e4ba7b8: Network error: Client connection negotiation failed: client connection to 127.3.12.65:40183: connect: Connection refused (error 111)
I20260805 23:09:20.955042 31854 catalog_manager.cc:4841] TS d66d1c5cc2c046988adc912fc9d0ffcd (127.3.12.67:43995): tablet b7ccb18bc623430aacdf7d036e4ba7b8 (table test-table [id=65a366f5721b4cac9ff082764e278079]) successfully deleted
I20260805 23:09:20.960490 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 32335
I20260805 23:09:20.969578 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 31833
2026-08-05T23:09:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1137 ms)
[----------] 6 tests from EncryptionPolicyTest (11454 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:21 dist-test-slave-2d19 krb5kdc[32600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:21 dist-test-slave-2d19 krb5kdc[32600](info): set up 2 sockets
Aug 05 23:09:21 dist-test-slave-2d19 krb5kdc[32600](info): commencing operation
krb5kdc: starting...
W20260805 23:09:22.202760 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:22 dist-test-slave-2d19 krb5kdc[32600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:22.337165 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:22.448925 32628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:22.449200 32628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:22.449241 32628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:22.453155 32628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:22.453214 32628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:22.453234 32628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:22.453251 32628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:22.453266 32628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:22.458528 32628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:22.459539 32628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:22.460610 32628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:22.469233 32640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:22.469240 32636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:22.469244 32637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:22.469506 32628 server_base.cc:1034] running on GCE node
I20260805 23:09:22.470166 32628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:22.471325 32628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:22.472512 32628 hybrid_clock.cc:648] HybridClock initialized: now 1785971362472486 us; error 41 us; skew 500 ppm
Aug 05 23:09:22 dist-test-slave-2d19 krb5kdc[32600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:22.475299 32628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:22.476357 32628 webserver.cc:460] Webserver started at http://127.0.0.1:38159/ using document root <none> and password file <none>
I20260805 23:09:22.476967 32628 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:22.477061 32628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:22.477300 32628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:22.479360 32628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "57ae9025b30743e8a03932b340992c7f"
format_stamp: "Formatted at 2026-08-05 23:09:22 on dist-test-slave-2d19"
server_key: "27c9a878ff3d9ff01203c82a5b951f46"
server_key_iv: "695a385596a8bc988361470dbda70aa7"
server_key_version: "encryptionkey@0"
I20260805 23:09:22.479938 32628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "57ae9025b30743e8a03932b340992c7f"
format_stamp: "Formatted at 2026-08-05 23:09:22 on dist-test-slave-2d19"
server_key: "27c9a878ff3d9ff01203c82a5b951f46"
server_key_iv: "695a385596a8bc988361470dbda70aa7"
server_key_version: "encryptionkey@0"
I20260805 23:09:22.483824 32628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:09:22.486340 32650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:22.487581 32628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:22.487726 32628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "57ae9025b30743e8a03932b340992c7f"
format_stamp: "Formatted at 2026-08-05 23:09:22 on dist-test-slave-2d19"
server_key: "27c9a878ff3d9ff01203c82a5b951f46"
server_key_iv: "695a385596a8bc988361470dbda70aa7"
server_key_version: "encryptionkey@0"
I20260805 23:09:22.487843 32628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:22.507553 32628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:22.510531 32628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:22.510752 32628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:22.518733 32628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36545
I20260805 23:09:22.518774 32732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36545 every 8 connection(s)
I20260805 23:09:22.519711 32628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:22.522599 32734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:22.523937 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 32628
I20260805 23:09:22.524088 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:22.524466 3121 external_mini_cluster.cc:1428] Setting key 0de38252d517b5da3829e20071bf356c
I20260805 23:09:22.529377 32734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f: Bootstrap starting.
I20260805 23:09:22.531723 32734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:22.532593 32734 log.cc:824] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:09:22 dist-test-slave-2d19 krb5kdc[32600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260805 23:09:22.535013 32734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f: No bootstrap required, opened a new log
I20260805 23:09:22.539389 32734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ae9025b30743e8a03932b340992c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260805 23:09:22.539549 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:22.526077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34015 (local address 127.0.0.1:36545)
0805 23:09:22.526611 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:22.526627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:22.526665 (+ 38us) server_negotiation.cc:407] Connection header received
0805 23:09:22.527299 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:22.527316 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:22.527680 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:22.528037 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:22.528925 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:22.529980 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:22.530879 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:22.531360 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:22.531536 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:22.534648 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:22.534653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:22.534668 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:22.534694 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:22.536497 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:22.537100 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:22.537105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:22.537110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:22.537219 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:22.537614 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:22.537617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:22.537618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:22.537985 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:22.538156 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:22.538495 (+ 339us) server_negotiation.cc:299] Negotiation successful
0805 23:09:22.538765 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":154,"threads_started":1}
I20260805 23:09:22.539620 32734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:22.539686 32734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ae9025b30743e8a03932b340992c7f, State: Initialized, Role: FOLLOWER
I20260805 23:09:22.540127 32734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ae9025b30743e8a03932b340992c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260805 23:09:22.540307 32734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:22.540431 32734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:22.540552 32734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:22.542346 32734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ae9025b30743e8a03932b340992c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260805 23:09:22.542667 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:22.542761 32734 leader_election.cc:302] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57ae9025b30743e8a03932b340992c7f; no voters:
I20260805 23:09:22.543099 32734 leader_election.cc:288] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:22.543215 32739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:22.543454 32739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [term 1 LEADER]: Becoming Leader. State: Replica: 57ae9025b30743e8a03932b340992c7f, State: Running, Role: LEADER
I20260805 23:09:22.543833 32739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ae9025b30743e8a03932b340992c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260805 23:09:22.544564 32734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:22.545822 32740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57ae9025b30743e8a03932b340992c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ae9025b30743e8a03932b340992c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260805 23:09:22.545964 32740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:22.546417 32761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:22.546399 32741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57ae9025b30743e8a03932b340992c7f. Latest consensus state: current_term: 1 leader_uuid: "57ae9025b30743e8a03932b340992c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ae9025b30743e8a03932b340992c7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260805 23:09:22.546506 32741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:22.549633 32761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:22.556205 32761 catalog_manager.cc:1347] Generated new cluster ID: 0dfea7eebb544e2683e6329c118bb514
I20260805 23:09:22.556377 32737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:22.545303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56745 (local address 127.0.0.1:36545)
0805 23:09:22.545417 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:22.545421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:22.547157 (+ 1736us) server_negotiation.cc:407] Connection header received
0805 23:09:22.547404 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:22.547406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:22.547450 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:22.547527 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:22.548411 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:22.549239 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:22.550351 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:22.550734 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:22.550849 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:22.552483 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:22.552493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:22.552496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:22.552519 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:22.554168 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:22.554737 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:22.554741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:22.554743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:22.554828 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:22.555239 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:22.555243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:22.555246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:22.555458 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:22.555575 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:22.556009 (+ 434us) server_negotiation.cc:299] Negotiation successful
0805 23:09:22.556128 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260805 23:09:22.556385 32761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:22.574399 32761 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:22.575368 32761 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:22.586946 32761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57ae9025b30743e8a03932b340992c7f: Generated new TSK 0
I20260805 23:09:22.587832 32761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:22.601449 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 32628
W20260805 23:09:22.614533 32598 connection.cc:535] client connection to 127.0.0.1:36545 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36545): Connection reset by peer (error 104)
2026-08-05T23:09:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1632 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:22 dist-test-slave-2d19 krb5kdc[309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:22 dist-test-slave-2d19 krb5kdc[309](info): set up 2 sockets
Aug 05 23:09:22 dist-test-slave-2d19 krb5kdc[309](info): commencing operation
krb5kdc: starting...
W20260805 23:09:23.826185 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:23 dist-test-slave-2d19 krb5kdc[309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:23.959077 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:24.074147 328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:24.074422 328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:24.074465 328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:24.078466 328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:24.078532 328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:24.078552 328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:24.078569 328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:24.078583 328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:24.083866 328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:24.084971 328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:24.085940 328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:24.094761 335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:24.094959 338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:24.095050 328 server_base.cc:1034] running on GCE node
W20260805 23:09:24.094777 336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:24.096006 328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:24.097550 328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:24.098768 328 hybrid_clock.cc:648] HybridClock initialized: now 1785971364098746 us; error 35 us; skew 500 ppm
Aug 05 23:09:24 dist-test-slave-2d19 krb5kdc[309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:24.106896 328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:24.107931 328 webserver.cc:460] Webserver started at http://127.0.0.1:43609/ using document root <none> and password file <none>
I20260805 23:09:24.108587 328 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:24.108654 328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:24.108902 328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:24.111009 328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "237328650eaa45988529cf89ad8d34db"
format_stamp: "Formatted at 2026-08-05 23:09:24 on dist-test-slave-2d19"
server_key: "45369cae465c8e33ff33633c2d959f73"
server_key_iv: "13b6ded998d8b28a924f2cb33c1d46fd"
server_key_version: "encryptionkey@0"
I20260805 23:09:24.111583 328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "237328650eaa45988529cf89ad8d34db"
format_stamp: "Formatted at 2026-08-05 23:09:24 on dist-test-slave-2d19"
server_key: "45369cae465c8e33ff33633c2d959f73"
server_key_iv: "13b6ded998d8b28a924f2cb33c1d46fd"
server_key_version: "encryptionkey@0"
I20260805 23:09:24.115417 328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:09:24.117887 351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:24.119129 328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:09:24.119267 328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "237328650eaa45988529cf89ad8d34db"
format_stamp: "Formatted at 2026-08-05 23:09:24 on dist-test-slave-2d19"
server_key: "45369cae465c8e33ff33633c2d959f73"
server_key_iv: "13b6ded998d8b28a924f2cb33c1d46fd"
server_key_version: "encryptionkey@0"
I20260805 23:09:24.119374 328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:24.131880 328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:24.134953 328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:24.135161 328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:24.142776 328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42351
I20260805 23:09:24.142802 431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42351 every 8 connection(s)
I20260805 23:09:24.143714 328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:24.146278 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 328
I20260805 23:09:24.146420 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:24.146622 432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:24.146785 3121 external_mini_cluster.cc:1428] Setting key 6f1cb6846c76a419d519491607bfb559
I20260805 23:09:24.153028 432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db: Bootstrap starting.
I20260805 23:09:24.155436 432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:24.155668 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:24.148189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33409 (local address 127.0.0.1:42351)
0805 23:09:24.148663 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:24.148679 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:24.148712 (+ 33us) server_negotiation.cc:407] Connection header received
0805 23:09:24.149359 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:24.149376 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:24.149701 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:24.150033 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:24.150433 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:24.151405 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:24.152399 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:24.152864 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:24.153047 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:24.153974 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:24.153979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:24.153991 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:24.154366 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:24.154417 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:24.154439 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:24.154584 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:24.154782 (+ 198us) server_negotiation.cc:299] Negotiation successful
0805 23:09:24.154921 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":150,"threads_started":1}
I20260805 23:09:24.156545 432 log.cc:824] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:24.158829 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:24.158833 432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db: No bootstrap required, opened a new log
I20260805 23:09:24.163167 432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "237328650eaa45988529cf89ad8d34db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42351 } }
I20260805 23:09:24.163475 432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:24.163569 432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237328650eaa45988529cf89ad8d34db, State: Initialized, Role: FOLLOWER
I20260805 23:09:24.164125 432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "237328650eaa45988529cf89ad8d34db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42351 } }
I20260805 23:09:24.164350 432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:24.164443 432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:24.164578 432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:24.165831 432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "237328650eaa45988529cf89ad8d34db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42351 } }
I20260805 23:09:24.166239 432 leader_election.cc:302] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 237328650eaa45988529cf89ad8d34db; no voters:
I20260805 23:09:24.166627 432 leader_election.cc:288] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:24.166750 444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:24.167004 444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [term 1 LEADER]: Becoming Leader. State: Replica: 237328650eaa45988529cf89ad8d34db, State: Running, Role: LEADER
I20260805 23:09:24.167411 444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "237328650eaa45988529cf89ad8d34db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42351 } }
I20260805 23:09:24.168205 432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:24.168689 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:24.160998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40859 (local address 127.0.0.1:42351)
0805 23:09:24.161118 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:24.161122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:24.163843 (+ 2721us) server_negotiation.cc:407] Connection header received
0805 23:09:24.163984 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:24.163986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:24.164025 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:24.164096 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:24.164517 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:24.165130 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:24.166358 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:24.166674 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:24.166819 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:24.168011 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:24.168016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:24.168019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:24.168034 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:24.168066 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:24.168080 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:24.168206 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:24.168493 (+ 287us) server_negotiation.cc:299] Negotiation successful
0805 23:09:24.168536 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:09:24.168767 444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 237328650eaa45988529cf89ad8d34db. Latest consensus state: current_term: 1 leader_uuid: "237328650eaa45988529cf89ad8d34db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "237328650eaa45988529cf89ad8d34db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42351 } } }
I20260805 23:09:24.168876 444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:24.169421 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "237328650eaa45988529cf89ad8d34db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "237328650eaa45988529cf89ad8d34db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42351 } } }
I20260805 23:09:24.169612 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:24.169871 454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:24.173027 454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:24.179822 454 catalog_manager.cc:1347] Generated new cluster ID: e3f85aee1da84e72b928f60769377dbd
I20260805 23:09:24.179935 454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:24.193164 454 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:24.194159 454 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:24.199777 454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 237328650eaa45988529cf89ad8d34db: Generated new TSK 0
I20260805 23:09:24.200493 454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:24.204620 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 328
W20260805 23:09:24.222021 307 connection.cc:535] client connection to 127.0.0.1:42351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42351): Connection reset by peer (error 104)
2026-08-05T23:09:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1607 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:24 dist-test-slave-2d19 krb5kdc[469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:24 dist-test-slave-2d19 krb5kdc[469](info): set up 2 sockets
Aug 05 23:09:24 dist-test-slave-2d19 krb5kdc[469](info): commencing operation
krb5kdc: starting...
W20260805 23:09:25.439919 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:25 dist-test-slave-2d19 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:25.576387 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:25.690800 488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:25.691145 488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:25.691242 488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:25.695345 488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:25.695452 488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:25.695492 488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:25.695529 488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:25.695577 488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:25.701027 488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:25.702174 488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:25.703227 488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:25.712108 494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:25.712117 493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:25.712117 496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:25.712349 488 server_base.cc:1034] running on GCE node
I20260805 23:09:25.712939 488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:25.714033 488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:25.715229 488 hybrid_clock.cc:648] HybridClock initialized: now 1785971365715215 us; error 26 us; skew 500 ppm
Aug 05 23:09:25 dist-test-slave-2d19 krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:25.718083 488 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:25.718957 488 webserver.cc:460] Webserver started at http://127.0.0.1:45337/ using document root <none> and password file <none>
I20260805 23:09:25.719463 488 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:25.719523 488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:25.719696 488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:25.721711 488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693"
format_stamp: "Formatted at 2026-08-05 23:09:25 on dist-test-slave-2d19"
server_key: "cc2ee7a151ab1c2f55dd7e3ee7ee70d8"
server_key_iv: "9e2a62a1d3c6fffd68d3b2873baadca6"
server_key_version: "encryptionkey@0"
I20260805 23:09:25.722198 488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693"
format_stamp: "Formatted at 2026-08-05 23:09:25 on dist-test-slave-2d19"
server_key: "cc2ee7a151ab1c2f55dd7e3ee7ee70d8"
server_key_iv: "9e2a62a1d3c6fffd68d3b2873baadca6"
server_key_version: "encryptionkey@0"
I20260805 23:09:25.725925 488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260805 23:09:25.728193 504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:25.729436 488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:25.729600 488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693"
format_stamp: "Formatted at 2026-08-05 23:09:25 on dist-test-slave-2d19"
server_key: "cc2ee7a151ab1c2f55dd7e3ee7ee70d8"
server_key_iv: "9e2a62a1d3c6fffd68d3b2873baadca6"
server_key_version: "encryptionkey@0"
I20260805 23:09:25.729698 488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:25.734416 488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:25.737445 488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:25.737641 488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:25.744926 488 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38353
I20260805 23:09:25.744998 573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38353 every 8 connection(s)
I20260805 23:09:25.745867 488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:25.748562 578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:25.753484 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 488
I20260805 23:09:25.753635 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:25.753921 3121 external_mini_cluster.cc:1428] Setting key e604cd8b7b8136057ff75414cdc45af2
I20260805 23:09:25.754161 578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693: Bootstrap starting.
I20260805 23:09:25.756705 578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:25.757697 578 log.cc:824] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:25.759156 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:25.755203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55935 (local address 127.0.0.1:38353)
0805 23:09:25.755634 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:25.755650 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:25.755686 (+ 36us) server_negotiation.cc:407] Connection header received
0805 23:09:25.756366 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:25.756384 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:25.756671 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:25.756997 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:25.757906 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:25.757911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:25.757922 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:25.757934 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:25.757988 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:25.758007 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:25.758124 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:25.758287 (+ 163us) server_negotiation.cc:299] Negotiation successful
0805 23:09:25.758447 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260805 23:09:25.759769 578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693: No bootstrap required, opened a new log
I20260805 23:09:25.762156 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:25.764556 578 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } }
I20260805 23:09:25.764848 578 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:25.764940 578 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d4b8d670fb454bb0a4c8c6db3e8693, State: Initialized, Role: FOLLOWER
I20260805 23:09:25.765426 578 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } }
I20260805 23:09:25.765579 578 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:25.765655 578 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:25.765755 578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:25.766368 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:25.764610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42431 (local address 127.0.0.1:38353)
0805 23:09:25.764744 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:25.764748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:25.764975 (+ 227us) server_negotiation.cc:407] Connection header received
0805 23:09:25.765132 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:25.765135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:25.765174 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:25.765257 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:25.765980 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:25.765983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:25.765985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:25.765996 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:25.766022 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:25.766040 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:25.766137 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:25.766213 (+ 76us) server_negotiation.cc:299] Negotiation successful
0805 23:09:25.766252 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260805 23:09:25.766924 578 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } }
I20260805 23:09:25.767289 578 leader_election.cc:302] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7d4b8d670fb454bb0a4c8c6db3e8693; no voters:
I20260805 23:09:25.767619 578 leader_election.cc:288] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:25.767750 590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:25.767990 590 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [term 1 LEADER]: Becoming Leader. State: Replica: e7d4b8d670fb454bb0a4c8c6db3e8693, State: Running, Role: LEADER
I20260805 23:09:25.768404 590 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } }
I20260805 23:09:25.768944 578 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:25.770114 592 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d4b8d670fb454bb0a4c8c6db3e8693. Latest consensus state: current_term: 1 leader_uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } } }
I20260805 23:09:25.770259 592 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:25.770653 601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:25.770613 591 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d4b8d670fb454bb0a4c8c6db3e8693" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38353 } } }
I20260805 23:09:25.770776 591 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:25.774055 601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:25.779748 601 catalog_manager.cc:1347] Generated new cluster ID: 349d7d7da07c4382873b23987e16d3fd
I20260805 23:09:25.779820 601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:25.787956 601 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:25.788301 601 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:25.800143 601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7d4b8d670fb454bb0a4c8c6db3e8693: Generated new TSK 0
I20260805 23:09:25.800860 601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:25.808811 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 488
2026-08-05T23:09:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1599 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:25 dist-test-slave-2d19 krb5kdc[617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:25 dist-test-slave-2d19 krb5kdc[617](info): set up 2 sockets
Aug 05 23:09:25 dist-test-slave-2d19 krb5kdc[617](info): commencing operation
krb5kdc: starting...
W20260805 23:09:27.101092 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:27 dist-test-slave-2d19 krb5kdc[617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:27.236130 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:27.350811 637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:27.351131 637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:27.351208 637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:27.355340 637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:27.355443 637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:27.355481 637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:27.355537 637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:27.355576 637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:27.360962 637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:27.362041 637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:27.363097 637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:27.372241 644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:27.372241 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:27.372303 645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:27.372460 637 server_base.cc:1034] running on GCE node
I20260805 23:09:27.373265 637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:27.374320 637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:27.375483 637 hybrid_clock.cc:648] HybridClock initialized: now 1785971367375456 us; error 38 us; skew 500 ppm
Aug 05 23:09:27 dist-test-slave-2d19 krb5kdc[617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:27.378345 637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:27.379196 637 webserver.cc:460] Webserver started at http://127.0.0.1:45279/ using document root <none> and password file <none>
I20260805 23:09:27.379714 637 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:27.379772 637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:27.379931 637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:27.382036 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "b7ecc01985144260bd586923c44cb649"
format_stamp: "Formatted at 2026-08-05 23:09:27 on dist-test-slave-2d19"
server_key: "88244a753b4d8caa136e5891cc6f26eb"
server_key_iv: "c3e03c9963d83882296df783c6340cc3"
server_key_version: "encryptionkey@0"
I20260805 23:09:27.382541 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "b7ecc01985144260bd586923c44cb649"
format_stamp: "Formatted at 2026-08-05 23:09:27 on dist-test-slave-2d19"
server_key: "88244a753b4d8caa136e5891cc6f26eb"
server_key_iv: "c3e03c9963d83882296df783c6340cc3"
server_key_version: "encryptionkey@0"
I20260805 23:09:27.386343 637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260805 23:09:27.388772 659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:27.390002 637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:27.390141 637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "b7ecc01985144260bd586923c44cb649"
format_stamp: "Formatted at 2026-08-05 23:09:27 on dist-test-slave-2d19"
server_key: "88244a753b4d8caa136e5891cc6f26eb"
server_key_iv: "c3e03c9963d83882296df783c6340cc3"
server_key_version: "encryptionkey@0"
I20260805 23:09:27.390246 637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:27.411962 637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:27.415417 637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:27.415638 637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:27.423331 637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37809
I20260805 23:09:27.423384 731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37809 every 8 connection(s)
I20260805 23:09:27.424378 637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:27.427161 732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:27.432698 732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649: Bootstrap starting.
I20260805 23:09:27.433723 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 637
I20260805 23:09:27.433902 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:27.434188 3121 external_mini_cluster.cc:1428] Setting key a20e605f1167a680394472bbe6450cc1
I20260805 23:09:27.435499 732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:27.436311 732 log.cc:824] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:27.438519 732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649: No bootstrap required, opened a new log
I20260805 23:09:27.443104 732 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ecc01985144260bd586923c44cb649" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } }
I20260805 23:09:27.443358 732 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:27.443439 732 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ecc01985144260bd586923c44cb649, State: Initialized, Role: FOLLOWER
Aug 05 23:09:27 dist-test-slave-2d19 krb5kdc[617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260805 23:09:27.443990 732 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ecc01985144260bd586923c44cb649" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } }
I20260805 23:09:27.444170 732 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:27.444265 732 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:27.444376 732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:27.445287 732 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ecc01985144260bd586923c44cb649" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } }
I20260805 23:09:27.445647 732 leader_election.cc:302] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7ecc01985144260bd586923c44cb649; no voters:
I20260805 23:09:27.445998 732 leader_election.cc:288] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:27.446070 739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:27.446276 739 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [term 1 LEADER]: Becoming Leader. State: Replica: b7ecc01985144260bd586923c44cb649, State: Running, Role: LEADER
I20260805 23:09:27.446576 739 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ecc01985144260bd586923c44cb649" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } }
I20260805 23:09:27.447319 732 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:27.447753 739 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7ecc01985144260bd586923c44cb649. Latest consensus state: current_term: 1 leader_uuid: "b7ecc01985144260bd586923c44cb649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ecc01985144260bd586923c44cb649" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } } }
I20260805 23:09:27.447868 739 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:27.448310 745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:27.448333 740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7ecc01985144260bd586923c44cb649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ecc01985144260bd586923c44cb649" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37809 } } }
I20260805 23:09:27.448547 740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:27.450341 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:27.435391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36645 (local address 127.0.0.1:37809)
0805 23:09:27.435877 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:27.435892 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:27.435927 (+ 35us) server_negotiation.cc:407] Connection header received
0805 23:09:27.436717 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:27.436742 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:27.437080 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:27.437356 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:27.438263 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:27.439309 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:27.440535 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:27.441001 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:27.441135 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:27.444180 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:27.444187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:27.444205 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:27.444244 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:27.446393 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:27.446942 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:27.446946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:27.446950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:27.447068 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:27.447360 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:27.447365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:27.447367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:27.447892 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:27.448909 (+ 1017us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:27.449086 (+ 177us) server_negotiation.cc:299] Negotiation successful
0805 23:09:27.449392 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":172,"threads_started":1}
I20260805 23:09:27.451612 745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:27.453428 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:27.457667 745 catalog_manager.cc:1347] Generated new cluster ID: d3985af11c8140278917cba192f857dc
I20260805 23:09:27.457729 745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:27.464005 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:27.455949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40025 (local address 127.0.0.1:37809)
0805 23:09:27.456074 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:27.456080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:27.456308 (+ 228us) server_negotiation.cc:407] Connection header received
0805 23:09:27.456491 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:27.456494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:27.456541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:27.456604 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:27.457506 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:27.458148 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:27.459082 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:27.459337 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:27.459447 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:27.460876 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:27.460879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:27.460882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:27.460906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:27.462363 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:27.462799 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:27.462802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:27.462804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:27.462884 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:27.463158 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:27.463163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:27.463164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:27.463335 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:27.463470 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:27.463711 (+ 241us) server_negotiation.cc:299] Negotiation successful
0805 23:09:27.463823 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260805 23:09:27.485786 745 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:27.486878 745 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:27.492686 745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7ecc01985144260bd586923c44cb649: Generated new TSK 0
I20260805 23:09:27.493371 745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:27.500149 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 637
2026-08-05T23:09:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1691 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:27 dist-test-slave-2d19 krb5kdc[775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:27 dist-test-slave-2d19 krb5kdc[775](info): set up 2 sockets
Aug 05 23:09:27 dist-test-slave-2d19 krb5kdc[775](info): commencing operation
krb5kdc: starting...
W20260805 23:09:28.743058 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:28 dist-test-slave-2d19 krb5kdc[775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:28.875562 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:28.987955 793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:28.988242 793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:28.988309 793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:28.992331 793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:28.992395 793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:28.992420 793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:28.992439 793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:28.992453 793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:28.998317 793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:28.999393 793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:29.000470 793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:29.009151 805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:29.009227 793 server_base.cc:1034] running on GCE node
W20260805 23:09:29.009146 801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:29.009146 800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:29.009888 793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:29.011015 793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:29.012187 793 hybrid_clock.cc:648] HybridClock initialized: now 1785971369012168 us; error 31 us; skew 500 ppm
Aug 05 23:09:29 dist-test-slave-2d19 krb5kdc[775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:29.014935 793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:29.015851 793 webserver.cc:460] Webserver started at http://127.0.0.1:45783/ using document root <none> and password file <none>
I20260805 23:09:29.016425 793 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:29.016498 793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:29.016737 793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:29.018832 793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "aa56f389c4384ea58dd98ed46bab7130"
format_stamp: "Formatted at 2026-08-05 23:09:29 on dist-test-slave-2d19"
server_key: "43174a65473a46653f6382f80f190231"
server_key_iv: "572f36818d87763ae803442c47772c6b"
server_key_version: "encryptionkey@0"
I20260805 23:09:29.019392 793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "aa56f389c4384ea58dd98ed46bab7130"
format_stamp: "Formatted at 2026-08-05 23:09:29 on dist-test-slave-2d19"
server_key: "43174a65473a46653f6382f80f190231"
server_key_iv: "572f36818d87763ae803442c47772c6b"
server_key_version: "encryptionkey@0"
I20260805 23:09:29.023190 793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:09:29.025759 822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:29.026991 793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:09:29.027127 793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "aa56f389c4384ea58dd98ed46bab7130"
format_stamp: "Formatted at 2026-08-05 23:09:29 on dist-test-slave-2d19"
server_key: "43174a65473a46653f6382f80f190231"
server_key_iv: "572f36818d87763ae803442c47772c6b"
server_key_version: "encryptionkey@0"
I20260805 23:09:29.027236 793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:29.042340 793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:29.045394 793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:29.045606 793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:29.052794 793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46131
I20260805 23:09:29.052848 891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46131 every 8 connection(s)
I20260805 23:09:29.053781 793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:29.056617 892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:29.062130 892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130: Bootstrap starting.
I20260805 23:09:29.063102 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 793
I20260805 23:09:29.063278 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:29.063591 3121 external_mini_cluster.cc:1428] Setting key 693d604f6d106c4f1549a8d22533281b
I20260805 23:09:29.065162 892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:29.065929 892 log.cc:824] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:29.067880 892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130: No bootstrap required, opened a new log
I20260805 23:09:29.071465 892 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa56f389c4384ea58dd98ed46bab7130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260805 23:09:29.071682 892 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:29.071745 892 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa56f389c4384ea58dd98ed46bab7130, State: Initialized, Role: FOLLOWER
I20260805 23:09:29.072292 892 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa56f389c4384ea58dd98ed46bab7130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260805 23:09:29.072464 892 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:29.072539 892 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:29.072649 892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:29.073303 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:29.064803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38433 (local address 127.0.0.1:46131)
0805 23:09:29.065343 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:29.065359 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:29.065398 (+ 39us) server_negotiation.cc:407] Connection header received
0805 23:09:29.066185 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:29.066205 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:29.066564 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:29.066921 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:29.067373 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:29.068613 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:29.069627 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:29.070164 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:29.070364 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:29.071292 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:29.071296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:29.071310 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:29.071753 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:29.071833 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:29.071869 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:29.071988 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:29.072209 (+ 221us) server_negotiation.cc:299] Negotiation successful
0805 23:09:29.072462 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":146,"threads_started":1}
I20260805 23:09:29.073606 892 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa56f389c4384ea58dd98ed46bab7130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260805 23:09:29.073945 892 leader_election.cc:302] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa56f389c4384ea58dd98ed46bab7130; no voters:
I20260805 23:09:29.074280 892 leader_election.cc:288] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:29.074396 898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:29.074589 898 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [term 1 LEADER]: Becoming Leader. State: Replica: aa56f389c4384ea58dd98ed46bab7130, State: Running, Role: LEADER
I20260805 23:09:29.074884 898 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa56f389c4384ea58dd98ed46bab7130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260805 23:09:29.075546 892 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:29.076089 899 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa56f389c4384ea58dd98ed46bab7130. Latest consensus state: current_term: 1 leader_uuid: "aa56f389c4384ea58dd98ed46bab7130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa56f389c4384ea58dd98ed46bab7130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } } }
I20260805 23:09:29.076218 899 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:29.076491 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:29.076537 900 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa56f389c4384ea58dd98ed46bab7130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa56f389c4384ea58dd98ed46bab7130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } } }
I20260805 23:09:29.076632 900 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:29.076659 903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:29.079934 903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:29.083200 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:29.078805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46039 (local address 127.0.0.1:46131)
0805 23:09:29.078893 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:29.078898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:29.079351 (+ 453us) server_negotiation.cc:407] Connection header received
0805 23:09:29.079491 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:29.079494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:29.079536 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:29.079629 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:29.079902 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:29.080701 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:29.081514 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:29.081759 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:29.081868 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:29.082748 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:29.082752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:29.082755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:29.082768 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:29.082794 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:29.082806 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:29.082850 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:29.083019 (+ 169us) server_negotiation.cc:299] Negotiation successful
0805 23:09:29.083074 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260805 23:09:29.086000 903 catalog_manager.cc:1347] Generated new cluster ID: 91dd14e656dc417a9a862f09838bc70c
I20260805 23:09:29.086081 903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:29.100770 903 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:29.102051 903 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:29.110507 903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa56f389c4384ea58dd98ed46bab7130: Generated new TSK 0
I20260805 23:09:29.111352 903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:29.121402 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 793
2026-08-05T23:09:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1622 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:29 dist-test-slave-2d19 krb5kdc[933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:29 dist-test-slave-2d19 krb5kdc[933](info): set up 2 sockets
Aug 05 23:09:29 dist-test-slave-2d19 krb5kdc[933](info): commencing operation
krb5kdc: starting...
W20260805 23:09:30.346276 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:30 dist-test-slave-2d19 krb5kdc[933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:30.476990 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:30.591126 954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:30.591392 954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:30.591435 954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:30.595608 954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:30.595675 954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:30.595695 954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:30.595711 954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:30.595726 954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:30.601109 954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:30.602185 954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:30.603289 954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:09:30.612444 954 server_base.cc:1034] running on GCE node
W20260805 23:09:30.612339 961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:30.612383 962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:30.612354 964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:30.613137 954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:30.614218 954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:30.615422 954 hybrid_clock.cc:648] HybridClock initialized: now 1785971370615408 us; error 31 us; skew 500 ppm
Aug 05 23:09:30 dist-test-slave-2d19 krb5kdc[933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:30.618229 954 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:30.619133 954 webserver.cc:460] Webserver started at http://127.0.0.1:37657/ using document root <none> and password file <none>
I20260805 23:09:30.619709 954 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:30.619804 954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:30.620042 954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:30.622296 954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "6a52e9d50d724361b6ed07323d938282"
format_stamp: "Formatted at 2026-08-05 23:09:30 on dist-test-slave-2d19"
server_key: "7a9d0434f7f3ac1620a2bce7572ca5e9"
server_key_iv: "0cd38734924c8d0f39eb78975cd476b7"
server_key_version: "encryptionkey@0"
I20260805 23:09:30.622891 954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "6a52e9d50d724361b6ed07323d938282"
format_stamp: "Formatted at 2026-08-05 23:09:30 on dist-test-slave-2d19"
server_key: "7a9d0434f7f3ac1620a2bce7572ca5e9"
server_key_iv: "0cd38734924c8d0f39eb78975cd476b7"
server_key_version: "encryptionkey@0"
I20260805 23:09:30.626701 954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260805 23:09:30.629208 976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:30.630453 954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260805 23:09:30.630620 954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "6a52e9d50d724361b6ed07323d938282"
format_stamp: "Formatted at 2026-08-05 23:09:30 on dist-test-slave-2d19"
server_key: "7a9d0434f7f3ac1620a2bce7572ca5e9"
server_key_iv: "0cd38734924c8d0f39eb78975cd476b7"
server_key_version: "encryptionkey@0"
I20260805 23:09:30.630745 954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:30.635480 954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:30.638628 954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:30.638823 954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:30.646140 954 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34135
I20260805 23:09:30.646183 1044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34135 every 8 connection(s)
I20260805 23:09:30.647073 954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:30.649852 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:30.653956 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 954
I20260805 23:09:30.654109 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:30.654381 3121 external_mini_cluster.cc:1428] Setting key 50b72e1eddd9863c0a8896cd7d068fc3
I20260805 23:09:30.655650 1045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282: Bootstrap starting.
I20260805 23:09:30.658115 1045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:30.658880 1045 log.cc:824] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:30.659750 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:30.655781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46963 (local address 127.0.0.1:34135)
0805 23:09:30.656239 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:30.656299 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:30.656347 (+ 48us) server_negotiation.cc:407] Connection header received
0805 23:09:30.656988 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:30.657006 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:30.657293 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:30.657648 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:30.658524 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:30.658528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:30.658540 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:30.658551 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:30.658613 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:30.658634 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:30.658754 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:30.658859 (+ 105us) server_negotiation.cc:299] Negotiation successful
0805 23:09:30.659027 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":145,"threads_started":1}
I20260805 23:09:30.661001 1045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282: No bootstrap required, opened a new log
I20260805 23:09:30.662695 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:30.665050 1045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a52e9d50d724361b6ed07323d938282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } }
I20260805 23:09:30.665385 1045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:30.665463 1045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a52e9d50d724361b6ed07323d938282, State: Initialized, Role: FOLLOWER
I20260805 23:09:30.665972 1045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a52e9d50d724361b6ed07323d938282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } }
I20260805 23:09:30.666162 1045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:30.666239 1045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:30.666342 1045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:30.666613 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:30.664861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60775 (local address 127.0.0.1:34135)
0805 23:09:30.664985 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:30.664989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:30.665150 (+ 161us) server_negotiation.cc:407] Connection header received
0805 23:09:30.665272 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:30.665274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:30.665304 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:30.665402 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:30.666186 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:30.666190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:30.666193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:30.666211 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:30.666257 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:30.666271 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:30.666361 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:30.666440 (+ 79us) server_negotiation.cc:299] Negotiation successful
0805 23:09:30.666480 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 23:09:30.667377 1045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a52e9d50d724361b6ed07323d938282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } }
I20260805 23:09:30.667755 1045 leader_election.cc:302] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a52e9d50d724361b6ed07323d938282; no voters:
I20260805 23:09:30.668067 1045 leader_election.cc:288] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:30.668184 1060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:30.668449 1060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [term 1 LEADER]: Becoming Leader. State: Replica: 6a52e9d50d724361b6ed07323d938282, State: Running, Role: LEADER
I20260805 23:09:30.668821 1060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a52e9d50d724361b6ed07323d938282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } }
I20260805 23:09:30.669356 1045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:30.670372 1061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a52e9d50d724361b6ed07323d938282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a52e9d50d724361b6ed07323d938282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } } }
I20260805 23:09:30.670677 1064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a52e9d50d724361b6ed07323d938282. Latest consensus state: current_term: 1 leader_uuid: "6a52e9d50d724361b6ed07323d938282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a52e9d50d724361b6ed07323d938282" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } } }
I20260805 23:09:30.670799 1061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:30.670864 1064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282 [sys.catalog]: This master's current role is: LEADER
W20260805 23:09:30.674124 1082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:30.674211 1082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:30.674292 1077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:30.675084 1077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:30.680537 1077 catalog_manager.cc:1347] Generated new cluster ID: 74bb5d4683804141b4f607389554ab49
I20260805 23:09:30.680608 1077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:30.691865 1077 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:30.692219 1077 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:30.696807 1077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a52e9d50d724361b6ed07323d938282: Generated new TSK 0
I20260805 23:09:30.697438 1077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:30.704492 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 954
2026-08-05T23:09:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1585 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:30 dist-test-slave-2d19 krb5kdc[1089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:30 dist-test-slave-2d19 krb5kdc[1089](info): set up 2 sockets
Aug 05 23:09:30 dist-test-slave-2d19 krb5kdc[1089](info): commencing operation
krb5kdc: starting...
W20260805 23:09:31.948908 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:32 dist-test-slave-2d19 krb5kdc[1089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:32.083758 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:32.198633 1107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:32.198892 1107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:32.198942 1107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:32.202905 1107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:32.202970 1107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:32.202989 1107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:32.203006 1107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:32.203020 1107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:32.208081 1107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:32.209169 1107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:32.210153 1107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:09:32.218885 1107 server_base.cc:1034] running on GCE node
W20260805 23:09:32.218793 1114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:32.218801 1115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:32.218793 1117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:32.219691 1107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:32.220840 1107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:32.222028 1107 hybrid_clock.cc:648] HybridClock initialized: now 1785971372222014 us; error 28 us; skew 500 ppm
Aug 05 23:09:32 dist-test-slave-2d19 krb5kdc[1089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:32.224820 1107 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:32.225708 1107 webserver.cc:460] Webserver started at http://127.0.0.1:41375/ using document root <none> and password file <none>
I20260805 23:09:32.226282 1107 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:32.226378 1107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:32.226620 1107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:32.228771 1107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8"
format_stamp: "Formatted at 2026-08-05 23:09:32 on dist-test-slave-2d19"
server_key: "7025c215082c5cf9b62daf656390997a"
server_key_iv: "46b06383fe7b9012f9f6311366f68d54"
server_key_version: "encryptionkey@0"
I20260805 23:09:32.229342 1107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8"
format_stamp: "Formatted at 2026-08-05 23:09:32 on dist-test-slave-2d19"
server_key: "7025c215082c5cf9b62daf656390997a"
server_key_iv: "46b06383fe7b9012f9f6311366f68d54"
server_key_version: "encryptionkey@0"
I20260805 23:09:32.233228 1107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260805 23:09:32.235764 1126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:32.237048 1107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:32.237211 1107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8"
format_stamp: "Formatted at 2026-08-05 23:09:32 on dist-test-slave-2d19"
server_key: "7025c215082c5cf9b62daf656390997a"
server_key_iv: "46b06383fe7b9012f9f6311366f68d54"
server_key_version: "encryptionkey@0"
I20260805 23:09:32.237329 1107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:32.264423 1107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:32.267366 1107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:32.267580 1107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:32.275143 1107 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43159
I20260805 23:09:32.275159 1208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43159 every 8 connection(s)
I20260805 23:09:32.276106 1107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:32.278903 1210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:32.280640 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 1107
I20260805 23:09:32.280789 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:32.281117 3121 external_mini_cluster.cc:1428] Setting key 5a0fe83f220676d39c07854f49bab350
I20260805 23:09:32.285545 1210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8: Bootstrap starting.
I20260805 23:09:32.287773 1210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:32.288621 1210 log.cc:824] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:09:32 dist-test-slave-2d19 krb5kdc[1089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260805 23:09:32.290890 1210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8: No bootstrap required, opened a new log
I20260805 23:09:32.294148 1210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } }
I20260805 23:09:32.294360 1210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:32.294442 1210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0478e03b2bd2413e9bdd6bfd5ac765d8, State: Initialized, Role: FOLLOWER
I20260805 23:09:32.294885 1210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } }
I20260805 23:09:32.295022 1210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:32.295113 1210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:32.295258 1210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:32.296358 1210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } }
I20260805 23:09:32.296720 1210 leader_election.cc:302] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0478e03b2bd2413e9bdd6bfd5ac765d8; no voters:
I20260805 23:09:32.296988 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:32.282682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:36865 (local address 127.0.0.1:43159)
0805 23:09:32.283149 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:32.283164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:32.283203 (+ 39us) server_negotiation.cc:407] Connection header received
0805 23:09:32.283853 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:32.283869 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:32.284207 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:32.284643 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:32.285525 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:32.286576 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:32.287454 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:32.287996 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:32.288169 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:32.291219 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:32.291224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:32.291238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:32.291265 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:32.293543 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:32.294080 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:32.294085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:32.294090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:32.294200 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:32.294578 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:32.294581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:32.294582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:32.295034 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:32.295300 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:32.295794 (+ 494us) server_negotiation.cc:299] Negotiation successful
0805 23:09:32.296022 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":126,"threads_started":1}
I20260805 23:09:32.297063 1210 leader_election.cc:288] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:32.297145 1215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:32.297365 1215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [term 1 LEADER]: Becoming Leader. State: Replica: 0478e03b2bd2413e9bdd6bfd5ac765d8, State: Running, Role: LEADER
I20260805 23:09:32.297729 1215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } }
I20260805 23:09:32.298321 1210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:32.299783 1218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0478e03b2bd2413e9bdd6bfd5ac765d8. Latest consensus state: current_term: 1 leader_uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } } }
I20260805 23:09:32.299783 1216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0478e03b2bd2413e9bdd6bfd5ac765d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43159 } } }
I20260805 23:09:32.299949 1218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:32.299949 1216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:32.300156 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:32.300347 1235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:32.303625 1235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:32.309895 1235 catalog_manager.cc:1347] Generated new cluster ID: af7cc165df28470d9083f277d6ac3ce8
I20260805 23:09:32.309959 1235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:32.310932 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:32.302458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:38425 (local address 127.0.0.1:43159)
0805 23:09:32.302575 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:32.302580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:32.302929 (+ 349us) server_negotiation.cc:407] Connection header received
0805 23:09:32.303086 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:32.303089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:32.303139 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:32.303221 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:32.303933 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:32.304535 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:32.305338 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:32.305599 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:32.305707 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:32.307412 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:32.307416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:32.307419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:32.307447 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:32.309037 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:32.309536 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:32.309544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:32.309545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:32.309617 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:32.310009 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:32.310012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:32.310014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:32.310218 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:32.310366 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:32.310684 (+ 318us) server_negotiation.cc:299] Negotiation successful
0805 23:09:32.310784 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260805 23:09:32.319272 1235 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:32.320174 1235 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:32.324071 1235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0478e03b2bd2413e9bdd6bfd5ac765d8: Generated new TSK 0
I20260805 23:09:32.324751 1235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:32.328567 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 1107
W20260805 23:09:32.349123 1086 connection.cc:535] client connection to 127.0.0.1:43159 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43159): Connection reset by peer (error 104)
2026-08-05T23:09:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:32 dist-test-slave-2d19 krb5kdc[1253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:32 dist-test-slave-2d19 krb5kdc[1253](info): set up 2 sockets
Aug 05 23:09:32 dist-test-slave-2d19 krb5kdc[1253](info): commencing operation
krb5kdc: starting...
W20260805 23:09:33.562196 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:33 dist-test-slave-2d19 krb5kdc[1253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:33.694991 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:33.807384 1269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:33.807690 1269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:33.807767 1269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:33.811743 1269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:33.811846 1269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:33.811884 1269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:33.811937 1269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:33.811970 1269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:33.817255 1269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:33.818331 1269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:33.819356 1269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:33.827984 1278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:33.828012 1276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:33.828012 1274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:33.828293 1269 server_base.cc:1034] running on GCE node
I20260805 23:09:33.828891 1269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:33.829990 1269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:33.831209 1269 hybrid_clock.cc:648] HybridClock initialized: now 1785971373831179 us; error 39 us; skew 500 ppm
Aug 05 23:09:33 dist-test-slave-2d19 krb5kdc[1253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:33.834002 1269 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:33.834885 1269 webserver.cc:460] Webserver started at http://127.0.0.1:35633/ using document root <none> and password file <none>
I20260805 23:09:33.835453 1269 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:33.835546 1269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:33.835803 1269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:33.837914 1269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "1be496033c23406591c685a501b26e87"
format_stamp: "Formatted at 2026-08-05 23:09:33 on dist-test-slave-2d19"
server_key: "a00cf9041514b904dc1d67bb1e9c962e"
server_key_iv: "6c4369f0c34e6f673ed3cfd7aefbfe00"
server_key_version: "encryptionkey@0"
I20260805 23:09:33.838497 1269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "1be496033c23406591c685a501b26e87"
format_stamp: "Formatted at 2026-08-05 23:09:33 on dist-test-slave-2d19"
server_key: "a00cf9041514b904dc1d67bb1e9c962e"
server_key_iv: "6c4369f0c34e6f673ed3cfd7aefbfe00"
server_key_version: "encryptionkey@0"
I20260805 23:09:33.842329 1269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:33.844782 1287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:33.846035 1269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:33.846182 1269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "1be496033c23406591c685a501b26e87"
format_stamp: "Formatted at 2026-08-05 23:09:33 on dist-test-slave-2d19"
server_key: "a00cf9041514b904dc1d67bb1e9c962e"
server_key_iv: "6c4369f0c34e6f673ed3cfd7aefbfe00"
server_key_version: "encryptionkey@0"
I20260805 23:09:33.846295 1269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:33.864181 1269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:33.867604 1269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:33.867852 1269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:33.875429 1269 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45343
I20260805 23:09:33.875442 1366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45343 every 8 connection(s)
I20260805 23:09:33.876499 1269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:33.879351 1367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:33.882087 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 1269
I20260805 23:09:33.882241 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:33.882510 3121 external_mini_cluster.cc:1428] Setting key 8a26d32e3f3e932ef6374d9134b6bc04
I20260805 23:09:33.885450 1367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87: Bootstrap starting.
I20260805 23:09:33.887686 1367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:33.888523 1367 log.cc:824] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:33.890503 1367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87: No bootstrap required, opened a new log
I20260805 23:09:33.891574 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:33.883987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:44955 (local address 127.0.0.1:45343)
0805 23:09:33.884479 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:33.884495 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:33.884531 (+ 36us) server_negotiation.cc:407] Connection header received
0805 23:09:33.885167 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:33.885184 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:33.885533 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:33.885912 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:33.886352 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:33.887335 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:33.888301 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:33.888819 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:33.889002 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:33.889915 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:33.889920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:33.889933 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:33.890317 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:33.890377 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:33.890399 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:33.890543 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:33.890753 (+ 210us) server_negotiation.cc:299] Negotiation successful
0805 23:09:33.890889 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":139,"threads_started":1}
I20260805 23:09:33.894488 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:33.894343 1367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be496033c23406591c685a501b26e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45343 } }
I20260805 23:09:33.894641 1367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:33.894731 1367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be496033c23406591c685a501b26e87, State: Initialized, Role: FOLLOWER
I20260805 23:09:33.895175 1367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be496033c23406591c685a501b26e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45343 } }
I20260805 23:09:33.895326 1367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:33.895396 1367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:33.895503 1367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:33.896622 1367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be496033c23406591c685a501b26e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45343 } }
I20260805 23:09:33.896999 1367 leader_election.cc:302] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be496033c23406591c685a501b26e87; no voters:
I20260805 23:09:33.897351 1367 leader_election.cc:288] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:33.897460 1378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:33.897775 1378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [term 1 LEADER]: Becoming Leader. State: Replica: 1be496033c23406591c685a501b26e87, State: Running, Role: LEADER
I20260805 23:09:33.898135 1378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be496033c23406591c685a501b26e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45343 } }
I20260805 23:09:33.898747 1367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:33.899300 1378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be496033c23406591c685a501b26e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be496033c23406591c685a501b26e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45343 } } }
I20260805 23:09:33.899497 1378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:33.899748 1381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be496033c23406591c685a501b26e87. Latest consensus state: current_term: 1 leader_uuid: "1be496033c23406591c685a501b26e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be496033c23406591c685a501b26e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45343 } } }
I20260805 23:09:33.899840 1381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:33.899966 1389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:33.902096 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:33.896746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:59373 (local address 127.0.0.1:45343)
0805 23:09:33.896894 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:33.896898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:33.897102 (+ 204us) server_negotiation.cc:407] Connection header received
0805 23:09:33.897228 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:33.897230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:33.897281 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:33.897406 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:33.897740 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:33.898500 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:33.899699 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:33.900054 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:33.900154 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:33.901396 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:33.901400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:33.901404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:33.901420 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:33.901452 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:33.901470 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:33.901563 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:33.901919 (+ 356us) server_negotiation.cc:299] Negotiation successful
0805 23:09:33.901955 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260805 23:09:33.903231 1389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:33.908396 1389 catalog_manager.cc:1347] Generated new cluster ID: dc7846e675e74a15850d53a64e595bfe
I20260805 23:09:33.908509 1389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:33.922552 1389 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:33.923467 1389 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:33.933472 1389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be496033c23406591c685a501b26e87: Generated new TSK 0
I20260805 23:09:33.934217 1389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:33.939435 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 1269
W20260805 23:09:33.952129 1249 connection.cc:535] client connection to 127.0.0.1:45343 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45343): Connection reset by peer (error 104)
2026-08-05T23:09:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1602 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:34 dist-test-slave-2d19 krb5kdc[1410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:34 dist-test-slave-2d19 krb5kdc[1410](info): set up 2 sockets
Aug 05 23:09:34 dist-test-slave-2d19 krb5kdc[1410](info): commencing operation
krb5kdc: starting...
W20260805 23:09:35.159411 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:35 dist-test-slave-2d19 krb5kdc[1410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:35.302969 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:35.415127 1427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:35.415409 1427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:35.415450 1427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:35.419384 1427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:35.419447 1427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:35.419466 1427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:35.419483 1427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:35.419498 1427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:35.424777 1427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:35.425868 1427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:35.426896 1427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:35.436507 1436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:35.436494 1439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:35.436518 1435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:35.436599 1427 server_base.cc:1034] running on GCE node
I20260805 23:09:35.437529 1427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:35.438740 1427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:35.439939 1427 hybrid_clock.cc:648] HybridClock initialized: now 1785971375439922 us; error 31 us; skew 500 ppm
Aug 05 23:09:35 dist-test-slave-2d19 krb5kdc[1410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:35.448902 1427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:35.450021 1427 webserver.cc:460] Webserver started at http://127.0.0.1:32953/ using document root <none> and password file <none>
I20260805 23:09:35.450632 1427 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:35.450726 1427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:35.450968 1427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:35.453182 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "1ae5dddf011044db8099cc4356610f16"
format_stamp: "Formatted at 2026-08-05 23:09:35 on dist-test-slave-2d19"
server_key: "73c4bc2f11d2d79a1bd82c692593bebd"
server_key_iv: "6d5fcdee4cf7b95cd0cbd359c677cb00"
server_key_version: "encryptionkey@0"
I20260805 23:09:35.453758 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "1ae5dddf011044db8099cc4356610f16"
format_stamp: "Formatted at 2026-08-05 23:09:35 on dist-test-slave-2d19"
server_key: "73c4bc2f11d2d79a1bd82c692593bebd"
server_key_iv: "6d5fcdee4cf7b95cd0cbd359c677cb00"
server_key_version: "encryptionkey@0"
I20260805 23:09:35.458101 1427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:35.460711 1446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:35.462055 1427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:09:35.462217 1427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "1ae5dddf011044db8099cc4356610f16"
format_stamp: "Formatted at 2026-08-05 23:09:35 on dist-test-slave-2d19"
server_key: "73c4bc2f11d2d79a1bd82c692593bebd"
server_key_iv: "6d5fcdee4cf7b95cd0cbd359c677cb00"
server_key_version: "encryptionkey@0"
I20260805 23:09:35.462342 1427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:35.467198 1427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:35.470472 1427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:35.470669 1427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:35.478441 1427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40063
I20260805 23:09:35.478469 1512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40063 every 8 connection(s)
I20260805 23:09:35.479374 1427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:35.480268 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 1427
I20260805 23:09:35.480427 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:35.480834 3121 external_mini_cluster.cc:1428] Setting key 59ee96053bf8fdb031f206430fb99497
I20260805 23:09:35.483119 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:35.486277 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:35.482367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:35173 (local address 127.0.0.1:40063)
0805 23:09:35.482849 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:35.482871 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:35.482915 (+ 44us) server_negotiation.cc:407] Connection header received
0805 23:09:35.483617 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:35.483634 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:35.483913 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:35.484299 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:35.485119 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:35.485124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:35.485135 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:35.485146 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:35.485198 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:35.485217 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:35.485369 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:35.485440 (+ 71us) server_negotiation.cc:299] Negotiation successful
0805 23:09:35.485602 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260805 23:09:35.489373 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:35.490142 1513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16: Bootstrap starting.
I20260805 23:09:35.493160 1513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:35.493432 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:35.491581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:54513 (local address 127.0.0.1:40063)
0805 23:09:35.491694 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:35.491698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:35.491921 (+ 223us) server_negotiation.cc:407] Connection header received
0805 23:09:35.492081 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:35.492084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:35.492118 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:35.492185 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:35.492963 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:35.492966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:35.492968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:35.492978 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:35.493002 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:35.493012 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:35.493096 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:35.493221 (+ 125us) server_negotiation.cc:299] Negotiation successful
0805 23:09:35.493284 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260805 23:09:35.494185 1513 log.cc:824] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:35.496409 1513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16: No bootstrap required, opened a new log
I20260805 23:09:35.500308 1513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae5dddf011044db8099cc4356610f16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40063 } }
I20260805 23:09:35.500639 1513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:35.500727 1513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae5dddf011044db8099cc4356610f16, State: Initialized, Role: FOLLOWER
I20260805 23:09:35.501188 1513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae5dddf011044db8099cc4356610f16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40063 } }
I20260805 23:09:35.501353 1513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:35.501453 1513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:35.501564 1513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:35.502568 1513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae5dddf011044db8099cc4356610f16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40063 } }
I20260805 23:09:35.502941 1513 leader_election.cc:302] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ae5dddf011044db8099cc4356610f16; no voters:
I20260805 23:09:35.503284 1513 leader_election.cc:288] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:35.503394 1529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:35.503695 1529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae5dddf011044db8099cc4356610f16, State: Running, Role: LEADER
I20260805 23:09:35.504055 1529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae5dddf011044db8099cc4356610f16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40063 } }
I20260805 23:09:35.504673 1513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:35.505892 1530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ae5dddf011044db8099cc4356610f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae5dddf011044db8099cc4356610f16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40063 } } }
I20260805 23:09:35.505892 1532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ae5dddf011044db8099cc4356610f16. Latest consensus state: current_term: 1 leader_uuid: "1ae5dddf011044db8099cc4356610f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae5dddf011044db8099cc4356610f16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40063 } } }
I20260805 23:09:35.506045 1532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:35.506045 1530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:35.506502 1539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:35.509764 1539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:35.516508 1539 catalog_manager.cc:1347] Generated new cluster ID: df66e32fe0a14227b008976c200081dd
I20260805 23:09:35.516603 1539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:35.553906 1539 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:35.554303 1539 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:35.561556 1539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ae5dddf011044db8099cc4356610f16: Generated new TSK 0
I20260805 23:09:35.562376 1539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:35.571026 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 1427
2026-08-05T23:09:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1634 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:35 dist-test-slave-2d19 krb5kdc[1554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:35 dist-test-slave-2d19 krb5kdc[1554](info): set up 2 sockets
Aug 05 23:09:35 dist-test-slave-2d19 krb5kdc[1554](info): commencing operation
krb5kdc: starting...
W20260805 23:09:36.800182 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:36 dist-test-slave-2d19 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:36.935948 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:37.050156 1577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:37.050504 1577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:37.050596 1577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:37.054774 1577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:37.054869 1577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:37.054908 1577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:37.054946 1577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:37.054986 1577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:37.060300 1577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:37.061452 1577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:37.062498 1577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:37.071142 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:37.071223 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:37.071579 1577 server_base.cc:1034] running on GCE node
W20260805 23:09:37.071239 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:37.072187 1577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:37.073400 1577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:37.074574 1577 hybrid_clock.cc:648] HybridClock initialized: now 1785971377074555 us; error 32 us; skew 500 ppm
Aug 05 23:09:37 dist-test-slave-2d19 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:37.077804 1577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:37.078802 1577 webserver.cc:460] Webserver started at http://127.0.0.1:41945/ using document root <none> and password file <none>
I20260805 23:09:37.079339 1577 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:37.079397 1577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:37.079581 1577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:37.081689 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "683827c45afa49c7a14aaad66d9071e0"
format_stamp: "Formatted at 2026-08-05 23:09:37 on dist-test-slave-2d19"
server_key: "5b7f68b94d47b1c99fbfe86118a2ff37"
server_key_iv: "960f25704b36d4e6fa7bd201f95010b4"
server_key_version: "encryptionkey@0"
I20260805 23:09:37.082224 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "683827c45afa49c7a14aaad66d9071e0"
format_stamp: "Formatted at 2026-08-05 23:09:37 on dist-test-slave-2d19"
server_key: "5b7f68b94d47b1c99fbfe86118a2ff37"
server_key_iv: "960f25704b36d4e6fa7bd201f95010b4"
server_key_version: "encryptionkey@0"
I20260805 23:09:37.086081 1577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260805 23:09:37.088644 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:37.090013 1577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:09:37.090126 1577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "683827c45afa49c7a14aaad66d9071e0"
format_stamp: "Formatted at 2026-08-05 23:09:37 on dist-test-slave-2d19"
server_key: "5b7f68b94d47b1c99fbfe86118a2ff37"
server_key_iv: "960f25704b36d4e6fa7bd201f95010b4"
server_key_version: "encryptionkey@0"
I20260805 23:09:37.090217 1577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:37.103098 1577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:37.106179 1577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:37.106338 1577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:37.113683 1577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39287
I20260805 23:09:37.113713 1665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39287 every 8 connection(s)
I20260805 23:09:37.114646 1577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:37.117483 1666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:37.123065 1666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0: Bootstrap starting.
I20260805 23:09:37.124552 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 1577
I20260805 23:09:37.124712 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:37.125005 3121 external_mini_cluster.cc:1428] Setting key 71554293676d9be3b595c24b3288d51d
I20260805 23:09:37.125908 1666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:37.126865 1666 log.cc:824] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:37.129142 1666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0: No bootstrap required, opened a new log
I20260805 23:09:37.133337 1666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683827c45afa49c7a14aaad66d9071e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } }
I20260805 23:09:37.133571 1666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:37.133607 1666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683827c45afa49c7a14aaad66d9071e0, State: Initialized, Role: FOLLOWER
I20260805 23:09:37.134040 1666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683827c45afa49c7a14aaad66d9071e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } }
I20260805 23:09:37.134196 1666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:37.134286 1666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:37.134416 1666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 0 FOLLOWER]: Advancing to term 1
Aug 05 23:09:37 dist-test-slave-2d19 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260805 23:09:37.135491 1666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683827c45afa49c7a14aaad66d9071e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } }
I20260805 23:09:37.135892 1666 leader_election.cc:302] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 683827c45afa49c7a14aaad66d9071e0; no voters:
I20260805 23:09:37.136299 1666 leader_election.cc:288] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:37.136699 1674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:37.136931 1674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [term 1 LEADER]: Becoming Leader. State: Replica: 683827c45afa49c7a14aaad66d9071e0, State: Running, Role: LEADER
I20260805 23:09:37.137290 1674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683827c45afa49c7a14aaad66d9071e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } }
I20260805 23:09:37.137734 1666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:37.139191 1673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "683827c45afa49c7a14aaad66d9071e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683827c45afa49c7a14aaad66d9071e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } } }
I20260805 23:09:37.139333 1673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:37.139722 1675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 683827c45afa49c7a14aaad66d9071e0. Latest consensus state: current_term: 1 leader_uuid: "683827c45afa49c7a14aaad66d9071e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683827c45afa49c7a14aaad66d9071e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39287 } } }
I20260805 23:09:37.139882 1675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:37.142189 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:37.126176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:37787 (local address 127.0.0.1:39287)
0805 23:09:37.126695 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:37.126715 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:37.126762 (+ 47us) server_negotiation.cc:407] Connection header received
0805 23:09:37.127566 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:37.127585 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:37.128042 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:37.128436 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:37.129104 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:37.130584 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:37.131462 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:37.132321 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:37.132489 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:37.135351 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:37.135360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:37.135379 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:37.135418 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:37.137712 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:37.138492 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:37.138503 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:37.138513 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:37.138682 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:37.139178 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:37.139182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:37.139183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:37.139782 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:37.140878 (+ 1096us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:37.141088 (+ 210us) server_negotiation.cc:299] Negotiation successful
0805 23:09:37.141280 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":154,"threads_started":1}
W20260805 23:09:37.143121 1690 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:37.143203 1690 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:37.144179 1683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:37.145342 1683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:37.145642 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:37.152294 1683 catalog_manager.cc:1347] Generated new cluster ID: 78470055565549348110be7797adf6a0
I20260805 23:09:37.152426 1683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:37.157828 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:37.148214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:56087 (local address 127.0.0.1:39287)
0805 23:09:37.148336 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:37.148341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:37.148605 (+ 264us) server_negotiation.cc:407] Connection header received
0805 23:09:37.148729 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:37.148732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:37.148779 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:37.148867 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:37.149744 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:37.150355 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:37.151765 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:37.152096 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:37.152177 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:37.153986 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:37.153989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:37.153992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:37.154026 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:37.155552 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:37.156101 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:37.156108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:37.156112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:37.156208 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:37.156700 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:37.156704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:37.156706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:37.156932 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:37.157061 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:37.157556 (+ 495us) server_negotiation.cc:299] Negotiation successful
0805 23:09:37.157661 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260805 23:09:37.167609 1683 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:37.168753 1683 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:37.174156 1683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 683827c45afa49c7a14aaad66d9071e0: Generated new TSK 0
I20260805 23:09:37.174888 1683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:37.182029 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 1577
W20260805 23:09:37.198319 1551 connection.cc:535] client connection to 127.0.0.1:39287 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39287): Connection reset by peer (error 104)
2026-08-05T23:09:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1611 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:37 dist-test-slave-2d19 krb5kdc[1705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:37 dist-test-slave-2d19 krb5kdc[1705](info): set up 2 sockets
Aug 05 23:09:37 dist-test-slave-2d19 krb5kdc[1705](info): commencing operation
krb5kdc: starting...
W20260805 23:09:38.430035 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:38 dist-test-slave-2d19 krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:38.568753 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:38.686411 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:38.686681 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:38.686724 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:38.690614 1720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:38.690672 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:38.690691 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:38.690707 1720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:38.690722 1720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:38.695996 1720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:38.697113 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:38.698092 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:38.706761 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:38.706761 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:38.706857 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:38.706982 1720 server_base.cc:1034] running on GCE node
I20260805 23:09:38.707692 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:38.708918 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:38.710083 1720 hybrid_clock.cc:648] HybridClock initialized: now 1785971378710064 us; error 33 us; skew 500 ppm
Aug 05 23:09:38 dist-test-slave-2d19 krb5kdc[1705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:38.713591 1720 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:38.714591 1720 webserver.cc:460] Webserver started at http://127.0.0.1:38453/ using document root <none> and password file <none>
I20260805 23:09:38.715260 1720 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:38.715333 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:38.715682 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:38.717909 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "d5ad7084517f440a992e2a400bca0046"
format_stamp: "Formatted at 2026-08-05 23:09:38 on dist-test-slave-2d19"
server_key: "82079c0b16a299a1454523d938a013cc"
server_key_iv: "2798e7f67faa59446a81b2843af94b2b"
server_key_version: "encryptionkey@0"
I20260805 23:09:38.718506 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "d5ad7084517f440a992e2a400bca0046"
format_stamp: "Formatted at 2026-08-05 23:09:38 on dist-test-slave-2d19"
server_key: "82079c0b16a299a1454523d938a013cc"
server_key_iv: "2798e7f67faa59446a81b2843af94b2b"
server_key_version: "encryptionkey@0"
I20260805 23:09:38.722502 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260805 23:09:38.725090 1743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:38.726398 1720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:09:38.726559 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "d5ad7084517f440a992e2a400bca0046"
format_stamp: "Formatted at 2026-08-05 23:09:38 on dist-test-slave-2d19"
server_key: "82079c0b16a299a1454523d938a013cc"
server_key_iv: "2798e7f67faa59446a81b2843af94b2b"
server_key_version: "encryptionkey@0"
I20260805 23:09:38.726670 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:38.745553 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:38.749408 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:38.749660 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:38.757457 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41825
I20260805 23:09:38.757486 1812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41825 every 8 connection(s)
I20260805 23:09:38.758471 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:38.761415 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:38.767119 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 1720
I20260805 23:09:38.767253 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:38.767395 1813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046: Bootstrap starting.
I20260805 23:09:38.767572 3121 external_mini_cluster.cc:1428] Setting key a82db6213c88b38b6f6f09f3128a39e6
I20260805 23:09:38.770069 1813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:38.770819 1813 log.cc:824] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:38.772799 1813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046: No bootstrap required, opened a new log
I20260805 23:09:38.776597 1813 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ad7084517f440a992e2a400bca0046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41825 } }
I20260805 23:09:38.776849 1813 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:38.776929 1813 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad7084517f440a992e2a400bca0046, State: Initialized, Role: FOLLOWER
I20260805 23:09:38.777077 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:38.769035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:41485 (local address 127.0.0.1:41825)
0805 23:09:38.769507 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:38.769523 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:38.769560 (+ 37us) server_negotiation.cc:407] Connection header received
0805 23:09:38.770275 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:38.770294 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:38.770666 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:38.771019 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:38.771471 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:38.772533 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:38.773513 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:38.773984 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:38.774171 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:38.775177 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:38.775182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:38.775195 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:38.775602 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:38.775659 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:38.775683 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:38.775829 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:38.776023 (+ 194us) server_negotiation.cc:299] Negotiation successful
0805 23:09:38.776167 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":128,"threads_started":1}
I20260805 23:09:38.777511 1813 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ad7084517f440a992e2a400bca0046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41825 } }
I20260805 23:09:38.777684 1813 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:38.777740 1813 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:38.777833 1813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:38.778790 1813 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ad7084517f440a992e2a400bca0046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41825 } }
I20260805 23:09:38.779170 1813 leader_election.cc:302] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ad7084517f440a992e2a400bca0046; no voters:
I20260805 23:09:38.779546 1813 leader_election.cc:288] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:38.779706 1820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:38.779958 1820 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [term 1 LEADER]: Becoming Leader. State: Replica: d5ad7084517f440a992e2a400bca0046, State: Running, Role: LEADER
I20260805 23:09:38.780380 1820 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ad7084517f440a992e2a400bca0046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41825 } }
I20260805 23:09:38.780512 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:38.780896 1813 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:38.782176 1823 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ad7084517f440a992e2a400bca0046. Latest consensus state: current_term: 1 leader_uuid: "d5ad7084517f440a992e2a400bca0046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ad7084517f440a992e2a400bca0046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41825 } } }
I20260805 23:09:38.782289 1823 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:38.782637 1821 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5ad7084517f440a992e2a400bca0046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ad7084517f440a992e2a400bca0046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41825 } } }
I20260805 23:09:38.782716 1836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:38.782745 1821 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:38.786450 1836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:38.788488 1818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:38.783395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:42713 (local address 127.0.0.1:41825)
0805 23:09:38.783532 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:38.783536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:38.783840 (+ 304us) server_negotiation.cc:407] Connection header received
0805 23:09:38.784071 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:38.784074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:38.784128 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:38.784211 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:38.784641 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:38.785394 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:38.786533 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:38.786884 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:38.786965 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:38.788022 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:38.788025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:38.788027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:38.788039 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:38.788069 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:38.788080 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:38.788155 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:38.788351 (+ 196us) server_negotiation.cc:299] Negotiation successful
0805 23:09:38.788384 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260805 23:09:38.792824 1836 catalog_manager.cc:1347] Generated new cluster ID: 62ab1449d1664b15b5e28423663c91ad
I20260805 23:09:38.792903 1836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:38.812974 1836 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:38.814352 1836 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:38.822782 1836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5ad7084517f440a992e2a400bca0046: Generated new TSK 0
I20260805 23:09:38.823486 1836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:38.833608 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 1720
W20260805 23:09:38.850600 1703 connection.cc:535] client connection to 127.0.0.1:41825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41825): Connection reset by peer (error 104)
2026-08-05T23:09:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1652 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:38 dist-test-slave-2d19 krb5kdc[1855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:38 dist-test-slave-2d19 krb5kdc[1855](info): set up 2 sockets
Aug 05 23:09:38 dist-test-slave-2d19 krb5kdc[1855](info): commencing operation
krb5kdc: starting...
W20260805 23:09:40.080796 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:40 dist-test-slave-2d19 krb5kdc[1855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260805 23:09:40.219563 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:40.334664 1872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:40.334955 1872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:40.335000 1872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:40.338917 1872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:40.338972 1872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:40.338992 1872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:40.339010 1872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:40.339025 1872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:40.344108 1872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:40.345187 1872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:40.346163 1872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:40.355015 1877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:40.355015 1881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:40.355015 1879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:40.355419 1872 server_base.cc:1034] running on GCE node
I20260805 23:09:40.355952 1872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:40.357190 1872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:40.358358 1872 hybrid_clock.cc:648] HybridClock initialized: now 1785971380358339 us; error 32 us; skew 500 ppm
Aug 05 23:09:40 dist-test-slave-2d19 krb5kdc[1855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:40.361263 1872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260805 23:09:40.362206 1872 webserver.cc:460] Webserver started at http://127.0.0.1:41417/ using document root <none> and password file <none>
I20260805 23:09:40.362869 1872 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:40.362934 1872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:40.363176 1872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:40.365444 1872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "f18bbd07acd4458f92fb093a5cbe8216"
format_stamp: "Formatted at 2026-08-05 23:09:40 on dist-test-slave-2d19"
server_key: "4a1a6a3f301edbdac9f790aa71d8b337"
server_key_iv: "79f8176c1bc4343a7933b4032a7053fe"
server_key_version: "encryptionkey@0"
I20260805 23:09:40.366030 1872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "f18bbd07acd4458f92fb093a5cbe8216"
format_stamp: "Formatted at 2026-08-05 23:09:40 on dist-test-slave-2d19"
server_key: "4a1a6a3f301edbdac9f790aa71d8b337"
server_key_iv: "79f8176c1bc4343a7933b4032a7053fe"
server_key_version: "encryptionkey@0"
I20260805 23:09:40.369952 1872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:40.372514 1891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:40.373797 1872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:40.373947 1872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "f18bbd07acd4458f92fb093a5cbe8216"
format_stamp: "Formatted at 2026-08-05 23:09:40 on dist-test-slave-2d19"
server_key: "4a1a6a3f301edbdac9f790aa71d8b337"
server_key_iv: "79f8176c1bc4343a7933b4032a7053fe"
server_key_version: "encryptionkey@0"
I20260805 23:09:40.374063 1872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:40.378784 1872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:40.382097 1872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:40.382305 1872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:40.389703 1872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46691
I20260805 23:09:40.389739 1974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46691 every 8 connection(s)
I20260805 23:09:40.390724 1872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:40.393683 1976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:40.397915 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 1872
I20260805 23:09:40.398080 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:40.398387 3121 external_mini_cluster.cc:1428] Setting key 603040151a34f1f0e3ddba805bf2991d
I20260805 23:09:40.399799 1976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216: Bootstrap starting.
I20260805 23:09:40.402642 1976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:40.403613 1976 log.cc:824] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:40.404366 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:40.399905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:35495 (local address 127.0.0.1:46691)
0805 23:09:40.400449 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:40.400467 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:40.400518 (+ 51us) server_negotiation.cc:407] Connection header received
0805 23:09:40.401384 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:40.401409 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:40.401779 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:40.402139 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:40.402977 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:40.402983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:40.403001 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:40.403013 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:40.403075 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:40.403101 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:40.403214 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:40.403305 (+ 91us) server_negotiation.cc:299] Negotiation successful
0805 23:09:40.403463 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":113,"threads_started":1}
I20260805 23:09:40.405875 1976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216: No bootstrap required, opened a new log
I20260805 23:09:40.407217 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:40.410071 1976 raft_consensus.cc:348] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18bbd07acd4458f92fb093a5cbe8216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260805 23:09:40.410369 1976 raft_consensus.cc:374] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:40.410441 1976 raft_consensus.cc:729] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18bbd07acd4458f92fb093a5cbe8216, State: Initialized, Role: FOLLOWER
I20260805 23:09:40.410939 1976 consensus_queue.cc:260] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18bbd07acd4458f92fb093a5cbe8216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260805 23:09:40.411139 1976 raft_consensus.cc:388] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:40.411146 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:40.409290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:44347 (local address 127.0.0.1:46691)
0805 23:09:40.409426 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:40.409431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:40.409659 (+ 228us) server_negotiation.cc:407] Connection header received
0805 23:09:40.409774 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:40.409776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:40.409806 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:40.409872 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:40.410666 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:40.410668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:40.410670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:40.410681 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:40.410707 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:40.410717 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:40.410789 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:40.410902 (+ 113us) server_negotiation.cc:299] Negotiation successful
0805 23:09:40.410933 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260805 23:09:40.411224 1976 raft_consensus.cc:482] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:40.411345 1976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:40.412513 1976 raft_consensus.cc:504] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18bbd07acd4458f92fb093a5cbe8216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260805 23:09:40.412858 1976 leader_election.cc:302] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f18bbd07acd4458f92fb093a5cbe8216; no voters:
I20260805 23:09:40.413169 1976 leader_election.cc:288] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:40.413313 1989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:40.413565 1989 raft_consensus.cc:686] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [term 1 LEADER]: Becoming Leader. State: Replica: f18bbd07acd4458f92fb093a5cbe8216, State: Running, Role: LEADER
I20260805 23:09:40.413915 1989 consensus_queue.cc:237] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18bbd07acd4458f92fb093a5cbe8216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } }
I20260805 23:09:40.414410 1976 sys_catalog.cc:564] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:40.415673 1993 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f18bbd07acd4458f92fb093a5cbe8216. Latest consensus state: current_term: 1 leader_uuid: "f18bbd07acd4458f92fb093a5cbe8216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18bbd07acd4458f92fb093a5cbe8216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } } }
I20260805 23:09:40.415665 1992 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f18bbd07acd4458f92fb093a5cbe8216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18bbd07acd4458f92fb093a5cbe8216" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46691 } } }
I20260805 23:09:40.415815 1993 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:40.415829 1992 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:40.416819 2003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:40.420181 2003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:40.426240 2003 catalog_manager.cc:1347] Generated new cluster ID: ff7e8e97631a4b9c95ebb4fc9b244b28
I20260805 23:09:40.426330 2003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:40.460515 2003 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:40.460884 2003 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:40.473570 2003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f18bbd07acd4458f92fb093a5cbe8216: Generated new TSK 0
I20260805 23:09:40.474450 2003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:40.494820 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 1872
2026-08-05T23:09:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1660 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:40 dist-test-slave-2d19 krb5kdc[2026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:40 dist-test-slave-2d19 krb5kdc[2026](info): set up 2 sockets
Aug 05 23:09:40 dist-test-slave-2d19 krb5kdc[2026](info): commencing operation
krb5kdc: starting...
W20260805 23:09:41.735773 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:41 dist-test-slave-2d19 krb5kdc[2026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:41.877417 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41611
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:41611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:41.991426 2050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:41.991684 2050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:41.991725 2050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:41.995720 2050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:41.995785 2050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:41.995805 2050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:41.995821 2050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:41.995836 2050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:42.001062 2050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:41611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:42.002192 2050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:42.003264 2050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:42.012458 2055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:42.012590 2050 server_base.cc:1034] running on GCE node
W20260805 23:09:42.012354 2056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:42.012362 2058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:42.013238 2050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:42.014464 2050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:42.015697 2050 hybrid_clock.cc:648] HybridClock initialized: now 1785971382015657 us; error 49 us; skew 500 ppm
Aug 05 23:09:42 dist-test-slave-2d19 krb5kdc[2026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971382, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:42.018724 2050 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:42.019596 2050 webserver.cc:460] Webserver started at http://127.3.12.126:45669/ using document root <none> and password file <none>
I20260805 23:09:42.020128 2050 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:42.020187 2050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:42.020428 2050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:42.022491 2050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "6c78a52baf844dceace9c7d6ecba9396"
format_stamp: "Formatted at 2026-08-05 23:09:42 on dist-test-slave-2d19"
server_key: "c281d30f1edec497b3d0e3d71f97bb49"
server_key_iv: "cf388e7eacaf0da466c5ea5c982a5cb2"
server_key_version: "encryptionkey@0"
I20260805 23:09:42.023025 2050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "6c78a52baf844dceace9c7d6ecba9396"
format_stamp: "Formatted at 2026-08-05 23:09:42 on dist-test-slave-2d19"
server_key: "c281d30f1edec497b3d0e3d71f97bb49"
server_key_iv: "cf388e7eacaf0da466c5ea5c982a5cb2"
server_key_version: "encryptionkey@0"
I20260805 23:09:42.027078 2050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 23:09:42.029692 2070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:42.030970 2050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:09:42.031086 2050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "6c78a52baf844dceace9c7d6ecba9396"
format_stamp: "Formatted at 2026-08-05 23:09:42 on dist-test-slave-2d19"
server_key: "c281d30f1edec497b3d0e3d71f97bb49"
server_key_iv: "cf388e7eacaf0da466c5ea5c982a5cb2"
server_key_version: "encryptionkey@0"
I20260805 23:09:42.031184 2050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:42.045576 2050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:42.048605 2050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:42.048763 2050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:42.056159 2050 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:41611
I20260805 23:09:42.056188 2152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:41611 every 8 connection(s)
I20260805 23:09:42.057138 2050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:42.059906 2153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:42.064990 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 2050
I20260805 23:09:42.065165 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:42.065447 3121 external_mini_cluster.cc:1428] Setting key e8abf92534f4eebd99fac9fd35bd9163
I20260805 23:09:42.065626 2153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396: Bootstrap starting.
I20260805 23:09:42.068200 2153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:42.069002 2153 log.cc:824] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:42.070942 2153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396: No bootstrap required, opened a new log
I20260805 23:09:42.074721 2153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c78a52baf844dceace9c7d6ecba9396" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41611 } }
Aug 05 23:09:42 dist-test-slave-2d19 krb5kdc[2026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:42.074976 2153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:42.075063 2153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c78a52baf844dceace9c7d6ecba9396, State: Initialized, Role: FOLLOWER
I20260805 23:09:42.075522 2153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c78a52baf844dceace9c7d6ecba9396" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41611 } }
I20260805 23:09:42.075721 2153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:42.075794 2153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:42.075912 2153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:42.077433 2153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c78a52baf844dceace9c7d6ecba9396" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41611 } }
I20260805 23:09:42.077818 2153 leader_election.cc:302] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c78a52baf844dceace9c7d6ecba9396; no voters:
I20260805 23:09:42.078249 2153 leader_election.cc:288] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:42.078333 2160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:42.078553 2160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [term 1 LEADER]: Becoming Leader. State: Replica: 6c78a52baf844dceace9c7d6ecba9396, State: Running, Role: LEADER
I20260805 23:09:42.078920 2160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c78a52baf844dceace9c7d6ecba9396" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41611 } }
I20260805 23:09:42.079623 2153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:42.080641 2165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c78a52baf844dceace9c7d6ecba9396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c78a52baf844dceace9c7d6ecba9396" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41611 } } }
I20260805 23:09:42.080773 2165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:42.081067 2164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c78a52baf844dceace9c7d6ecba9396. Latest consensus state: current_term: 1 leader_uuid: "6c78a52baf844dceace9c7d6ecba9396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c78a52baf844dceace9c7d6ecba9396" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 41611 } } }
I20260805 23:09:42.081218 2164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:42.081961 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:42.066906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40685 (local address 127.3.12.126:41611)
0805 23:09:42.067433 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:42.067455 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:42.067499 (+ 44us) server_negotiation.cc:407] Connection header received
0805 23:09:42.068512 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:42.068531 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:42.068937 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:42.069327 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:42.070199 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:42.071250 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:42.072138 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:42.072695 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:42.072868 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:42.075702 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:42.075710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:42.075729 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:42.075770 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:42.077911 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:42.078666 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:42.078672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:42.078678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:42.078826 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:42.079263 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:42.079266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:42.079268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:42.079806 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:42.080042 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:42.080472 (+ 430us) server_negotiation.cc:299] Negotiation successful
0805 23:09:42.080751 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":131,"threads_started":1}
W20260805 23:09:42.085019 2182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:42.085108 2182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:42.085184 2175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:42.085513 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:42.085983 2175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:42.091611 2175 catalog_manager.cc:1347] Generated new cluster ID: 06c25035342a43f69637c138d42191f1
I20260805 23:09:42.091679 2175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:42.098273 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:42.087770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58011 (local address 127.3.12.126:41611)
0805 23:09:42.087887 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:42.087891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:42.089609 (+ 1718us) server_negotiation.cc:407] Connection header received
0805 23:09:42.089765 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:42.089768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:42.089817 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:42.089882 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:42.090727 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:42.091323 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:42.092485 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:42.092845 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:42.092975 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:42.094631 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:42.094635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:42.094638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:42.094665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:42.096390 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:42.096908 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:42.096912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:42.096914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:42.097013 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:42.097345 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:42.097348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:42.097350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:42.097581 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:42.097732 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:42.098001 (+ 269us) server_negotiation.cc:299] Negotiation successful
0805 23:09:42.098105 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:09:42.108560 2175 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:42.109858 2175 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:42.115419 2175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c78a52baf844dceace9c7d6ecba9396: Generated new TSK 0
I20260805 23:09:42.116308 2175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:42.127385 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 2050
W20260805 23:09:42.140793 2024 connection.cc:535] client connection to 127.3.12.126:41611 recv error: Network error: failed to read from TLS socket (remote: 127.3.12.126:41611): Connection reset by peer (error 104)
2026-08-05T23:09:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:42 dist-test-slave-2d19 krb5kdc[2199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:42 dist-test-slave-2d19 krb5kdc[2199](info): set up 2 sockets
Aug 05 23:09:42 dist-test-slave-2d19 krb5kdc[2199](info): commencing operation
krb5kdc: starting...
W20260805 23:09:43.355178 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:43 dist-test-slave-2d19 krb5kdc[2199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:43.491580 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:33793
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:33793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:43.607551 2218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:43.607879 2218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:43.607969 2218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:43.612736 2218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:43.612828 2218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:43.612869 2218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:43.612907 2218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:43.612943 2218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:43.618384 2218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:33793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:33793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:43.619575 2218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:43.620718 2218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:43.629702 2231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:43.629702 2227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:43.629698 2226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:43.629742 2218 server_base.cc:1034] running on GCE node
I20260805 23:09:43.630491 2218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:43.631669 2218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:43.632838 2218 hybrid_clock.cc:648] HybridClock initialized: now 1785971383632793 us; error 56 us; skew 500 ppm
Aug 05 23:09:43 dist-test-slave-2d19 krb5kdc[2199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:43.635885 2218 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:43.636900 2218 webserver.cc:460] Webserver started at http://127.3.12.126:43895/ using document root <none> and password file <none>
I20260805 23:09:43.637490 2218 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:43.637588 2218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:43.637835 2218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:43.639916 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "cd67d76d75c84d8bb337e8add85bd6d2"
format_stamp: "Formatted at 2026-08-05 23:09:43 on dist-test-slave-2d19"
server_key: "548015e65eeff77722f64b5f9f800e8f"
server_key_iv: "aac2471103b2d24008e9ea6eaeca2676"
server_key_version: "encryptionkey@0"
I20260805 23:09:43.640525 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "cd67d76d75c84d8bb337e8add85bd6d2"
format_stamp: "Formatted at 2026-08-05 23:09:43 on dist-test-slave-2d19"
server_key: "548015e65eeff77722f64b5f9f800e8f"
server_key_iv: "aac2471103b2d24008e9ea6eaeca2676"
server_key_version: "encryptionkey@0"
I20260805 23:09:43.644304 2218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260805 23:09:43.646708 2239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:43.647989 2218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:43.648145 2218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "cd67d76d75c84d8bb337e8add85bd6d2"
format_stamp: "Formatted at 2026-08-05 23:09:43 on dist-test-slave-2d19"
server_key: "548015e65eeff77722f64b5f9f800e8f"
server_key_iv: "aac2471103b2d24008e9ea6eaeca2676"
server_key_version: "encryptionkey@0"
I20260805 23:09:43.648260 2218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:43.667869 2218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:43.671029 2218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:43.671248 2218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:43.679155 2218 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:33793
I20260805 23:09:43.679175 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:33793 every 8 connection(s)
I20260805 23:09:43.680178 2218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:43.683136 2326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:43.688838 2326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2: Bootstrap starting.
I20260805 23:09:43.689566 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 2218
I20260805 23:09:43.689714 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:43.690023 3121 external_mini_cluster.cc:1428] Setting key 7eaa3fcc74c5dd5d08dc6175b5aa24a5
I20260805 23:09:43.691946 2326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:43.692775 2326 log.cc:824] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:43.695128 2326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2: No bootstrap required, opened a new log
I20260805 23:09:43.699676 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:43.691391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45855 (local address 127.3.12.126:33793)
0805 23:09:43.691862 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:43.691878 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:43.691931 (+ 53us) server_negotiation.cc:407] Connection header received
0805 23:09:43.692686 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:43.692705 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:43.693111 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:43.693520 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:43.693950 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:43.694927 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:43.696294 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:43.696767 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:43.696891 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:43.697875 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:43.697880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:43.697892 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:43.698296 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:43.698351 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:43.698374 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:43.698478 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:43.698737 (+ 259us) server_negotiation.cc:299] Negotiation successful
0805 23:09:43.698892 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":137,"threads_started":1}
I20260805 23:09:43.699870 2326 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd67d76d75c84d8bb337e8add85bd6d2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33793 } }
I20260805 23:09:43.700174 2326 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:43.700284 2326 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67d76d75c84d8bb337e8add85bd6d2, State: Initialized, Role: FOLLOWER
I20260805 23:09:43.700765 2326 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd67d76d75c84d8bb337e8add85bd6d2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33793 } }
I20260805 23:09:43.700955 2326 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:43.701047 2326 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:43.701188 2326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:43.702291 2326 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd67d76d75c84d8bb337e8add85bd6d2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33793 } }
I20260805 23:09:43.702665 2326 leader_election.cc:302] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd67d76d75c84d8bb337e8add85bd6d2; no voters:
I20260805 23:09:43.702890 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:43.703073 2326 leader_election.cc:288] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:43.703169 2333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:43.703388 2333 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [term 1 LEADER]: Becoming Leader. State: Replica: cd67d76d75c84d8bb337e8add85bd6d2, State: Running, Role: LEADER
I20260805 23:09:43.703743 2333 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd67d76d75c84d8bb337e8add85bd6d2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33793 } }
I20260805 23:09:43.704587 2326 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:43.705626 2334 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd67d76d75c84d8bb337e8add85bd6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd67d76d75c84d8bb337e8add85bd6d2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33793 } } }
I20260805 23:09:43.705780 2334 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:43.706261 2350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:43.705626 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd67d76d75c84d8bb337e8add85bd6d2. Latest consensus state: current_term: 1 leader_uuid: "cd67d76d75c84d8bb337e8add85bd6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd67d76d75c84d8bb337e8add85bd6d2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 33793 } } }
I20260805 23:09:43.706535 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:43.709648 2350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:43.712131 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:43.706372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43293 (local address 127.3.12.126:33793)
0805 23:09:43.706475 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:43.706480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:43.706496 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:09:43.706682 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:43.706690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:43.706754 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:43.706819 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:43.707394 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:43.708337 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:43.709706 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:43.710082 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:43.710178 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:43.711543 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:43.711547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:43.711553 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:43.711572 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:43.711603 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:43.711618 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:43.711689 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:43.711969 (+ 280us) server_negotiation.cc:299] Negotiation successful
0805 23:09:43.712006 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260805 23:09:43.715821 2350 catalog_manager.cc:1347] Generated new cluster ID: 5bbade25640e4ad3ac3dbd44a9cdfb3f
I20260805 23:09:43.715893 2350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:43.733079 2350 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:43.734380 2350 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:43.741035 2350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd67d76d75c84d8bb337e8add85bd6d2: Generated new TSK 0
I20260805 23:09:43.741884 2350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:43.749547 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 2218
W20260805 23:09:43.762276 2196 connection.cc:535] client connection to 127.3.12.126:33793 recv error: Network error: failed to read from TLS socket (remote: 127.3.12.126:33793): Connection reset by peer (error 104)
2026-08-05T23:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1621 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:43 dist-test-slave-2d19 krb5kdc[2369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:43 dist-test-slave-2d19 krb5kdc[2369](info): set up 2 sockets
Aug 05 23:09:43 dist-test-slave-2d19 krb5kdc[2369](info): commencing operation
krb5kdc: starting...
W20260805 23:09:44.991458 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:45 dist-test-slave-2d19 krb5kdc[2369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:45.127908 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:35789
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:40811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:35789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:45.254479 2389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:45.254750 2389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:45.254794 2389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:45.258908 2389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:45.258975 2389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:45.258993 2389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:45.259011 2389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:45.259025 2389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:45.264410 2389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:40811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:35789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:35789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:45.265398 2389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:45.266345 2389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:45.274811 2402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:45.274847 2404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:45.274816 2401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:45.274823 2389 server_base.cc:1034] running on GCE node
I20260805 23:09:45.275774 2389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:45.276925 2389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:45.278136 2389 hybrid_clock.cc:648] HybridClock initialized: now 1785971385278116 us; error 33 us; skew 500 ppm
Aug 05 23:09:45 dist-test-slave-2d19 krb5kdc[2369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971385, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:45.280984 2389 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:45.281831 2389 webserver.cc:460] Webserver started at http://127.3.12.126:37829/ using document root <none> and password file <none>
I20260805 23:09:45.282395 2389 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:45.282503 2389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:45.282742 2389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:45.284826 2389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5"
format_stamp: "Formatted at 2026-08-05 23:09:45 on dist-test-slave-2d19"
server_key: "9482630ddf23c1c2192aa88fdbe4619a"
server_key_iv: "5ffcaec43aabaf9491ec8fd3431e06e9"
server_key_version: "encryptionkey@0"
I20260805 23:09:45.285386 2389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5"
format_stamp: "Formatted at 2026-08-05 23:09:45 on dist-test-slave-2d19"
server_key: "9482630ddf23c1c2192aa88fdbe4619a"
server_key_iv: "5ffcaec43aabaf9491ec8fd3431e06e9"
server_key_version: "encryptionkey@0"
I20260805 23:09:45.289157 2389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:09:45.291498 2417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:45.292760 2389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:45.292899 2389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5"
format_stamp: "Formatted at 2026-08-05 23:09:45 on dist-test-slave-2d19"
server_key: "9482630ddf23c1c2192aa88fdbe4619a"
server_key_iv: "5ffcaec43aabaf9491ec8fd3431e06e9"
server_key_version: "encryptionkey@0"
I20260805 23:09:45.293010 2389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:45.297477 2389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:45.300741 2389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:45.300938 2389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:45.307933 2389 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:35789
I20260805 23:09:45.307991 2490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:35789 every 8 connection(s)
I20260805 23:09:45.308950 2389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:45.311694 2491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:45.315083 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 2389
I20260805 23:09:45.315246 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:45.315527 3121 external_mini_cluster.cc:1428] Setting key bea84927f509ebe8330082a5f1ce4bb0
I20260805 23:09:45.317813 2491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5: Bootstrap starting.
I20260805 23:09:45.320446 2491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:45.320932 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:45.316936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55423 (local address 127.3.12.126:35789)
0805 23:09:45.317366 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:45.317382 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:45.317420 (+ 38us) server_negotiation.cc:407] Connection header received
0805 23:09:45.318113 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:45.318131 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:45.318416 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:45.318700 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:45.319572 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:45.319576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:45.319590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:45.319603 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:45.319658 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:45.319678 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:45.319789 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:45.319927 (+ 138us) server_negotiation.cc:299] Negotiation successful
0805 23:09:45.320131 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":137,"threads_started":1}
I20260805 23:09:45.321245 2491 log.cc:824] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:45.323364 2491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5: No bootstrap required, opened a new log
I20260805 23:09:45.323709 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:45.327455 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:45.325797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46693 (local address 127.3.12.126:35789)
0805 23:09:45.325914 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:45.325924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:45.326054 (+ 130us) server_negotiation.cc:407] Connection header received
0805 23:09:45.326186 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:45.326188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:45.326215 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:45.326278 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:45.327032 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:45.327034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:45.327036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:45.327046 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:45.327069 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:45.327081 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:45.327157 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:45.327277 (+ 120us) server_negotiation.cc:299] Negotiation successful
0805 23:09:45.327308 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260805 23:09:45.327759 2491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 35789 } }
I20260805 23:09:45.328032 2491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:45.328125 2491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccd4709c7b84a5fa80bbe70ec9913e5, State: Initialized, Role: FOLLOWER
I20260805 23:09:45.328655 2491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 35789 } }
I20260805 23:09:45.328809 2491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:45.328881 2491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:45.328976 2491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:45.329900 2491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 35789 } }
I20260805 23:09:45.330256 2491 leader_election.cc:302] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ccd4709c7b84a5fa80bbe70ec9913e5; no voters:
I20260805 23:09:45.330574 2491 leader_election.cc:288] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:45.330694 2508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:45.330932 2508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [term 1 LEADER]: Becoming Leader. State: Replica: 7ccd4709c7b84a5fa80bbe70ec9913e5, State: Running, Role: LEADER
I20260805 23:09:45.331254 2508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 35789 } }
I20260805 23:09:45.331789 2491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:45.333069 2511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccd4709c7b84a5fa80bbe70ec9913e5. Latest consensus state: current_term: 1 leader_uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 35789 } } }
I20260805 23:09:45.333200 2511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:45.333488 2510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccd4709c7b84a5fa80bbe70ec9913e5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 35789 } } }
I20260805 23:09:45.333565 2510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:45.334007 2519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:45.337074 2519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:45.342803 2519 catalog_manager.cc:1347] Generated new cluster ID: 05e4286230794700a82297eb652a7a8c
I20260805 23:09:45.342877 2519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:45.366569 2519 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:45.366930 2519 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:45.377883 2519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ccd4709c7b84a5fa80bbe70ec9913e5: Generated new TSK 0
I20260805 23:09:45.378618 2519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:45.381335 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 2389
2026-08-05T23:09:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1634 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:45 dist-test-slave-2d19 krb5kdc[2533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:45 dist-test-slave-2d19 krb5kdc[2533](info): set up 2 sockets
Aug 05 23:09:45 dist-test-slave-2d19 krb5kdc[2533](info): commencing operation
krb5kdc: starting...
W20260805 23:09:46.658046 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:46 dist-test-slave-2d19 krb5kdc[2533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:46.795861 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42737
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:42737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:46.909580 2553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:46.909909 2553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:46.909986 2553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:46.914137 2553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:46.914242 2553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:46.914283 2553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:46.914335 2553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:46.914371 2553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:46.919804 2553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:42737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:42737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:46.920985 2553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:46.922082 2553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:46.930861 2565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:46.930877 2562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:46.930879 2563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:46.931217 2553 server_base.cc:1034] running on GCE node
I20260805 23:09:46.931877 2553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:46.933113 2553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:46.934329 2553 hybrid_clock.cc:648] HybridClock initialized: now 1785971386934309 us; error 37 us; skew 500 ppm
Aug 05 23:09:46 dist-test-slave-2d19 krb5kdc[2533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971386, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:46.944188 2553 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:46.945322 2553 webserver.cc:460] Webserver started at http://127.3.12.126:45397/ using document root <none> and password file <none>
I20260805 23:09:46.945957 2553 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:46.946055 2553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:46.946302 2553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:46.948657 2553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "d247ab350bbd4bf9a17ec1ae92dd45de"
format_stamp: "Formatted at 2026-08-05 23:09:46 on dist-test-slave-2d19"
server_key: "be5ab3ba81a3581ad72ebd28d32d0fe0"
server_key_iv: "1c11614ee38054d278254d758443ecde"
server_key_version: "encryptionkey@0"
I20260805 23:09:46.949268 2553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "d247ab350bbd4bf9a17ec1ae92dd45de"
format_stamp: "Formatted at 2026-08-05 23:09:46 on dist-test-slave-2d19"
server_key: "be5ab3ba81a3581ad72ebd28d32d0fe0"
server_key_iv: "1c11614ee38054d278254d758443ecde"
server_key_version: "encryptionkey@0"
I20260805 23:09:46.953428 2553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260805 23:09:46.956053 2572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:46.957382 2553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:46.957562 2553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "d247ab350bbd4bf9a17ec1ae92dd45de"
format_stamp: "Formatted at 2026-08-05 23:09:46 on dist-test-slave-2d19"
server_key: "be5ab3ba81a3581ad72ebd28d32d0fe0"
server_key_iv: "1c11614ee38054d278254d758443ecde"
server_key_version: "encryptionkey@0"
I20260805 23:09:46.957687 2553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:46.970868 2553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:46.974049 2553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:46.974257 2553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:46.981689 2553 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:42737
I20260805 23:09:46.981711 2639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:42737 every 8 connection(s)
I20260805 23:09:46.982647 2553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:46.983307 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 2553
I20260805 23:09:46.983464 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:46.983749 3121 external_mini_cluster.cc:1428] Setting key 94709990ab897230fd049702f90725ca
I20260805 23:09:46.985925 2640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:46.992744 2640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de: Bootstrap starting.
Aug 05 23:09:46 dist-test-slave-2d19 krb5kdc[2533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:46.995169 2640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:46.995918 2640 log.cc:824] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:46.998170 2640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de: No bootstrap required, opened a new log
I20260805 23:09:46.998473 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:46.985162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52663 (local address 127.3.12.126:42737)
0805 23:09:46.985607 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:46.985623 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:46.985660 (+ 37us) server_negotiation.cc:407] Connection header received
0805 23:09:46.986365 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:46.986384 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:46.986726 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:46.987004 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:46.987921 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:46.989011 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:46.990185 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:46.990636 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:46.990747 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:46.993652 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:46.993657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:46.993670 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:46.993697 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:46.995429 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:46.995890 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:46.995895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:46.995900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:46.996037 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:46.996420 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:46.996423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:46.996425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:46.996945 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:46.997092 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:46.997408 (+ 316us) server_negotiation.cc:299] Negotiation successful
0805 23:09:46.997733 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":143,"threads_started":1}
I20260805 23:09:47.001653 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:47.002447 2640 raft_consensus.cc:348] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d247ab350bbd4bf9a17ec1ae92dd45de" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42737 } }
I20260805 23:09:47.002749 2640 raft_consensus.cc:374] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:47.002823 2640 raft_consensus.cc:729] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247ab350bbd4bf9a17ec1ae92dd45de, State: Initialized, Role: FOLLOWER
I20260805 23:09:47.003340 2640 consensus_queue.cc:260] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d247ab350bbd4bf9a17ec1ae92dd45de" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42737 } }
I20260805 23:09:47.003507 2640 raft_consensus.cc:388] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:47.003592 2640 raft_consensus.cc:482] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:47.003727 2640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:47.004972 2640 raft_consensus.cc:504] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d247ab350bbd4bf9a17ec1ae92dd45de" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42737 } }
I20260805 23:09:47.005371 2640 leader_election.cc:302] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d247ab350bbd4bf9a17ec1ae92dd45de; no voters:
I20260805 23:09:47.005751 2640 leader_election.cc:288] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:47.005875 2652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:47.006142 2652 raft_consensus.cc:686] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [term 1 LEADER]: Becoming Leader. State: Replica: d247ab350bbd4bf9a17ec1ae92dd45de, State: Running, Role: LEADER
I20260805 23:09:47.006533 2652 consensus_queue.cc:237] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d247ab350bbd4bf9a17ec1ae92dd45de" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42737 } }
I20260805 23:09:47.007100 2640 sys_catalog.cc:564] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:47.007825 2652 sys_catalog.cc:455] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [sys.catalog]: SysCatalogTable state changed. Reason: New leader d247ab350bbd4bf9a17ec1ae92dd45de. Latest consensus state: current_term: 1 leader_uuid: "d247ab350bbd4bf9a17ec1ae92dd45de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d247ab350bbd4bf9a17ec1ae92dd45de" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42737 } } }
I20260805 23:09:47.007966 2652 sys_catalog.cc:458] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:47.008522 2655 sys_catalog.cc:455] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d247ab350bbd4bf9a17ec1ae92dd45de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d247ab350bbd4bf9a17ec1ae92dd45de" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 42737 } } }
I20260805 23:09:47.008713 2655 sys_catalog.cc:458] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:47.009012 2667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:47.012742 2667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:47.013908 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:47.003947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35805 (local address 127.3.12.126:42737)
0805 23:09:47.004090 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:47.004095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:47.004344 (+ 249us) server_negotiation.cc:407] Connection header received
0805 23:09:47.004475 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:47.004478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:47.004527 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:47.004604 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:47.005282 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:47.006052 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:47.007166 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:47.007574 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:47.007696 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:47.009591 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:47.009595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:47.009598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:47.009627 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:47.011449 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:47.011975 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:47.011979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:47.011981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:47.012076 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:47.012442 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:47.012446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:47.012448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:47.012701 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:47.012841 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:47.013615 (+ 774us) server_negotiation.cc:299] Negotiation successful
0805 23:09:47.013748 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260805 23:09:47.018843 2667 catalog_manager.cc:1347] Generated new cluster ID: eccf7cc804214f44bce372891f6b8dba
I20260805 23:09:47.018920 2667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:47.029801 2667 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:47.030903 2667 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:47.048457 2667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d247ab350bbd4bf9a17ec1ae92dd45de: Generated new TSK 0
I20260805 23:09:47.049259 2667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:47.058108 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 2553
2026-08-05T23:09:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1671 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:47 dist-test-slave-2d19 krb5kdc[2686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:47 dist-test-slave-2d19 krb5kdc[2686](info): set up 2 sockets
Aug 05 23:09:47 dist-test-slave-2d19 krb5kdc[2686](info): commencing operation
krb5kdc: starting...
W20260805 23:09:48.297071 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:48 dist-test-slave-2d19 krb5kdc[2686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:48.432087 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36193
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:48.546411 2705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:48.546707 2705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:48.546751 2705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:48.550782 2705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:48.550854 2705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:48.550874 2705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:48.550891 2705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:48.550906 2705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:48.556192 2705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:36193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:48.557315 2705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:48.558306 2705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:48.567121 2714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:48.567150 2712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:48.567159 2711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:48.567190 2705 server_base.cc:1034] running on GCE node
I20260805 23:09:48.568194 2705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:48.569461 2705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:48.570672 2705 hybrid_clock.cc:648] HybridClock initialized: now 1785971388570653 us; error 31 us; skew 500 ppm
Aug 05 23:09:48 dist-test-slave-2d19 krb5kdc[2686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971388, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:48.573818 2705 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:48.574798 2705 webserver.cc:460] Webserver started at http://127.3.12.126:34219/ using document root <none> and password file <none>
I20260805 23:09:48.575395 2705 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:48.575462 2705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:48.575697 2705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:48.577937 2705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "cb9b65e382da4f01841d23e7a0b2a481"
format_stamp: "Formatted at 2026-08-05 23:09:48 on dist-test-slave-2d19"
server_key: "b3c39e10ad33a7a52d40c81f21f13560"
server_key_iv: "cc9c37a67753f5ba97293f014a401749"
server_key_version: "encryptionkey@0"
I20260805 23:09:48.578600 2705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "cb9b65e382da4f01841d23e7a0b2a481"
format_stamp: "Formatted at 2026-08-05 23:09:48 on dist-test-slave-2d19"
server_key: "b3c39e10ad33a7a52d40c81f21f13560"
server_key_iv: "cc9c37a67753f5ba97293f014a401749"
server_key_version: "encryptionkey@0"
I20260805 23:09:48.582662 2705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260805 23:09:48.585450 2721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:48.586670 2705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:09:48.586817 2705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "cb9b65e382da4f01841d23e7a0b2a481"
format_stamp: "Formatted at 2026-08-05 23:09:48 on dist-test-slave-2d19"
server_key: "b3c39e10ad33a7a52d40c81f21f13560"
server_key_iv: "cc9c37a67753f5ba97293f014a401749"
server_key_version: "encryptionkey@0"
I20260805 23:09:48.586921 2705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:48.620217 2705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:48.623674 2705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:48.623903 2705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:48.631557 2705 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:36193
I20260805 23:09:48.631582 2790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:36193 every 8 connection(s)
I20260805 23:09:48.632640 2705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:48.635511 2791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:48.639410 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 2705
I20260805 23:09:48.639592 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:48.639880 3121 external_mini_cluster.cc:1428] Setting key 99e9b43a87198d8f076ae2350bdb1f4a
I20260805 23:09:48.641659 2791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481: Bootstrap starting.
I20260805 23:09:48.643935 2791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:48.644783 2791 log.cc:824] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:48.646853 2791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481: No bootstrap required, opened a new log
I20260805 23:09:48.649598 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:48.641415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41589 (local address 127.3.12.126:36193)
0805 23:09:48.641923 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:48.641940 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:48.642006 (+ 66us) server_negotiation.cc:407] Connection header received
0805 23:09:48.642669 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:48.642686 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:48.643044 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:48.643397 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:48.643823 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:48.644987 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:48.646088 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:48.646561 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:48.646739 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:48.647798 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:48.647803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:48.647816 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:48.648220 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:48.648347 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:48.648375 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:48.648479 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:48.648649 (+ 170us) server_negotiation.cc:299] Negotiation successful
0805 23:09:48.648866 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":139,"threads_started":1}
I20260805 23:09:48.651032 2791 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9b65e382da4f01841d23e7a0b2a481" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36193 } }
I20260805 23:09:48.651279 2791 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:48.651368 2791 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9b65e382da4f01841d23e7a0b2a481, State: Initialized, Role: FOLLOWER
I20260805 23:09:48.651883 2791 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9b65e382da4f01841d23e7a0b2a481" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36193 } }
I20260805 23:09:48.652066 2791 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:48.652135 2791 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:48.652235 2791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:48.652781 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:48.653424 2791 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9b65e382da4f01841d23e7a0b2a481" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36193 } }
I20260805 23:09:48.653795 2791 leader_election.cc:302] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb9b65e382da4f01841d23e7a0b2a481; no voters:
I20260805 23:09:48.654146 2791 leader_election.cc:288] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:48.654268 2798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:48.654542 2798 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [term 1 LEADER]: Becoming Leader. State: Replica: cb9b65e382da4f01841d23e7a0b2a481, State: Running, Role: LEADER
I20260805 23:09:48.654923 2798 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9b65e382da4f01841d23e7a0b2a481" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36193 } }
I20260805 23:09:48.655776 2791 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:48.656759 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb9b65e382da4f01841d23e7a0b2a481. Latest consensus state: current_term: 1 leader_uuid: "cb9b65e382da4f01841d23e7a0b2a481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9b65e382da4f01841d23e7a0b2a481" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36193 } } }
I20260805 23:09:48.656881 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:48.657318 2804 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb9b65e382da4f01841d23e7a0b2a481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb9b65e382da4f01841d23e7a0b2a481" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 36193 } } }
I20260805 23:09:48.657460 2804 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:48.659906 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:48.655399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55391 (local address 127.3.12.126:36193)
0805 23:09:48.655523 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:48.655528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:48.655541 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:09:48.655578 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:48.655581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:48.655624 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:48.655711 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:48.656024 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:48.656896 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:48.658108 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:48.658477 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:48.658592 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:48.659468 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:48.659472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:48.659475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:48.659488 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:48.659519 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:48.659532 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:48.659610 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:48.659720 (+ 110us) server_negotiation.cc:299] Negotiation successful
0805 23:09:48.659779 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260805 23:09:48.661387 2827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:48.661480 2827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:48.661558 2819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:48.662258 2819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:48.667590 2819 catalog_manager.cc:1347] Generated new cluster ID: ac0a40e382004730aacc08125f37b7c3
I20260805 23:09:48.667668 2819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:48.681423 2819 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:48.682811 2819 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:48.691798 2819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb9b65e382da4f01841d23e7a0b2a481: Generated new TSK 0
I20260805 23:09:48.692695 2819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:48.696949 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 2705
2026-08-05T23:09:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1642 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:48 dist-test-slave-2d19 krb5kdc[2834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:48 dist-test-slave-2d19 krb5kdc[2834](info): set up 2 sockets
Aug 05 23:09:48 dist-test-slave-2d19 krb5kdc[2834](info): commencing operation
krb5kdc: starting...
W20260805 23:09:49.964200 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:50 dist-test-slave-2d19 krb5kdc[2834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:50.097847 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:39593
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:39593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:50.211433 2854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:50.211735 2854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:50.211815 2854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:50.215896 2854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:50.215991 2854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:50.216025 2854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:50.216063 2854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:50.216095 2854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:50.221542 2854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:39593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:39593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:50.222667 2854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:50.223731 2854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:50.232731 2860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:50.232744 2864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:50.232725 2861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:50.232904 2854 server_base.cc:1034] running on GCE node
I20260805 23:09:50.233537 2854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:50.234699 2854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:50.235864 2854 hybrid_clock.cc:648] HybridClock initialized: now 1785971390235826 us; error 54 us; skew 500 ppm
Aug 05 23:09:50 dist-test-slave-2d19 krb5kdc[2834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971390, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:50.238893 2854 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:50.239876 2854 webserver.cc:460] Webserver started at http://127.3.12.126:45937/ using document root <none> and password file <none>
I20260805 23:09:50.240523 2854 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:50.240592 2854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:50.240813 2854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:50.242897 2854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "44ddbaf718404bec9f5d68ee28dcadb5"
format_stamp: "Formatted at 2026-08-05 23:09:50 on dist-test-slave-2d19"
server_key: "10c74d01182df80dadf397608ae59cc1"
server_key_iv: "823072bdc30e6c42f3e157b009a151b0"
server_key_version: "encryptionkey@0"
I20260805 23:09:50.243470 2854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "44ddbaf718404bec9f5d68ee28dcadb5"
format_stamp: "Formatted at 2026-08-05 23:09:50 on dist-test-slave-2d19"
server_key: "10c74d01182df80dadf397608ae59cc1"
server_key_iv: "823072bdc30e6c42f3e157b009a151b0"
server_key_version: "encryptionkey@0"
I20260805 23:09:50.247324 2854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 23:09:50.249828 2872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:50.251073 2854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260805 23:09:50.251217 2854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "44ddbaf718404bec9f5d68ee28dcadb5"
format_stamp: "Formatted at 2026-08-05 23:09:50 on dist-test-slave-2d19"
server_key: "10c74d01182df80dadf397608ae59cc1"
server_key_iv: "823072bdc30e6c42f3e157b009a151b0"
server_key_version: "encryptionkey@0"
I20260805 23:09:50.251322 2854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:50.255924 2854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:50.259161 2854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:50.259354 2854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:50.266335 2854 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:39593
I20260805 23:09:50.266351 2948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:39593 every 8 connection(s)
I20260805 23:09:50.267251 2854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:50.270133 2949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:50.274902 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 2854
I20260805 23:09:50.275063 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:50.275354 3121 external_mini_cluster.cc:1428] Setting key 3aed672b3207d22787d9bd4aa0cfb6eb
I20260805 23:09:50.275924 2949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5: Bootstrap starting.
I20260805 23:09:50.278491 2949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:50.279389 2949 log.cc:824] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:50.280704 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:50.276852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58477 (local address 127.3.12.126:39593)
0805 23:09:50.277299 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:50.277313 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:50.277348 (+ 35us) server_negotiation.cc:407] Connection header received
0805 23:09:50.277966 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:50.277982 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:50.278325 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:50.278666 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:50.279451 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:50.279456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:50.279467 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:50.279477 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:50.279554 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:50.279580 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:50.279702 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:50.279813 (+ 111us) server_negotiation.cc:299] Negotiation successful
0805 23:09:50.279969 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":134,"threads_started":1}
I20260805 23:09:50.281471 2949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5: No bootstrap required, opened a new log
I20260805 23:09:50.283299 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:50.285373 2949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ddbaf718404bec9f5d68ee28dcadb5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39593 } }
I20260805 23:09:50.285678 2949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 23:09:50.285763 2949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ddbaf718404bec9f5d68ee28dcadb5, State: Initialized, Role: FOLLOWER
I20260805 23:09:50.286243 2949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ddbaf718404bec9f5d68ee28dcadb5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39593 } }
I20260805 23:09:50.286422 2949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 23:09:50.286505 2949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 23:09:50.286628 2949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:50.287189 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:50.285371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56615 (local address 127.3.12.126:39593)
0805 23:09:50.285488 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:50.285492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:50.285790 (+ 298us) server_negotiation.cc:407] Connection header received
0805 23:09:50.285924 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:50.285927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:50.285956 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:50.286032 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:50.286793 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:50.286796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:50.286799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0805 23:09:50.286810 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:50.286836 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0805 23:09:50.286852 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:50.286938 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:50.287025 (+ 87us) server_negotiation.cc:299] Negotiation successful
0805 23:09:50.287063 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260805 23:09:50.287724 2949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ddbaf718404bec9f5d68ee28dcadb5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39593 } }
I20260805 23:09:50.288129 2949 leader_election.cc:302] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44ddbaf718404bec9f5d68ee28dcadb5; no voters:
I20260805 23:09:50.288477 2949 leader_election.cc:288] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 23:09:50.288585 2963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:50.288822 2963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [term 1 LEADER]: Becoming Leader. State: Replica: 44ddbaf718404bec9f5d68ee28dcadb5, State: Running, Role: LEADER
I20260805 23:09:50.289160 2963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ddbaf718404bec9f5d68ee28dcadb5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39593 } }
I20260805 23:09:50.289710 2949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:50.290735 2964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44ddbaf718404bec9f5d68ee28dcadb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ddbaf718404bec9f5d68ee28dcadb5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39593 } } }
I20260805 23:09:50.290860 2964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:50.291204 2965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44ddbaf718404bec9f5d68ee28dcadb5. Latest consensus state: current_term: 1 leader_uuid: "44ddbaf718404bec9f5d68ee28dcadb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44ddbaf718404bec9f5d68ee28dcadb5" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 39593 } } }
I20260805 23:09:50.291360 2965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:50.291260 2974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:50.294456 2974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:50.300230 2974 catalog_manager.cc:1347] Generated new cluster ID: 4483bfdbd41346f8bc64f611fc886d78
I20260805 23:09:50.300323 2974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:50.314209 2974 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:50.314496 2974 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:50.320626 2974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44ddbaf718404bec9f5d68ee28dcadb5: Generated new TSK 0
I20260805 23:09:50.321405 2974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 23:09:50.325484 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 2854
2026-08-05T23:09:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1627 ms)
[----------] 18 tests from AuthTokenIssuingTest (29356 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:50 dist-test-slave-2d19 krb5kdc[2989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:50 dist-test-slave-2d19 krb5kdc[2989](info): set up 2 sockets
Aug 05 23:09:50 dist-test-slave-2d19 krb5kdc[2989](info): commencing operation
krb5kdc: starting...
W20260805 23:09:51.584282 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:51 dist-test-slave-2d19 krb5kdc[2989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:51.719219 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43225
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:43225,127.3.12.125:43203,127.3.12.124:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:51.839008 3009 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 23:09:51.839342 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:51.839411 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:51.839496 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:51.843487 3009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:51.843572 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:51.843612 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:51.843665 3009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:51.843705 3009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:51.849061 3009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:43225,127.3.12.125:43203,127.3.12.124:33789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:51.850123 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:51.851143 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260805 23:09:51.860128 3009 server_base.cc:1034] running on GCE node
W20260805 23:09:51.860076 3020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:51.860086 3018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:51.860075 3017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:51.860996 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:51.862074 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:51.863265 3009 hybrid_clock.cc:648] HybridClock initialized: now 1785971391863243 us; error 34 us; skew 500 ppm
Aug 05 23:09:51 dist-test-slave-2d19 krb5kdc[2989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:51.866153 3009 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:51.866988 3009 webserver.cc:460] Webserver started at http://127.3.12.126:44141/ using document root <none> and password file <none>
I20260805 23:09:51.867542 3009 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:51.867630 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:51.867853 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:51.869889 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2"
format_stamp: "Formatted at 2026-08-05 23:09:51 on dist-test-slave-2d19"
server_key: "6067009449ae76e5c31bc635a055d485"
server_key_iv: "7a8b52cb8efb4a72541a86847f62b446"
server_key_version: "encryptionkey@0"
I20260805 23:09:51.870440 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2"
format_stamp: "Formatted at 2026-08-05 23:09:51 on dist-test-slave-2d19"
server_key: "6067009449ae76e5c31bc635a055d485"
server_key_iv: "7a8b52cb8efb4a72541a86847f62b446"
server_key_version: "encryptionkey@0"
I20260805 23:09:51.874158 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260805 23:09:51.876581 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:51.877727 3009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 23:09:51.877864 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2"
format_stamp: "Formatted at 2026-08-05 23:09:51 on dist-test-slave-2d19"
server_key: "6067009449ae76e5c31bc635a055d485"
server_key_iv: "7a8b52cb8efb4a72541a86847f62b446"
server_key_version: "encryptionkey@0"
I20260805 23:09:51.877974 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:51.897866 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:51.900815 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:51.901021 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:51.908018 3009 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:43225
I20260805 23:09:51.908329 3106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:43225 every 8 connection(s)
I20260805 23:09:51.909042 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:51.911746 3107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:51.915450 3107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:51.916464 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3009
I20260805 23:09:51.916626 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:51.916926 3121 external_mini_cluster.cc:1428] Setting key 4a4d2abe63845ccfe931ec1f8a7ffeaf
Aug 05 23:09:51 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.125@KRBTEST.COM" created.
I20260805 23:09:51.932035 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:51.917856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.3.12.126:43225)
0805 23:09:51.918419 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:51.918433 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:51.918533 (+ 100us) server_negotiation.cc:407] Connection header received
0805 23:09:51.919961 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:51.919965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:51.920151 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:51.920359 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:51.921107 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:51.921966 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:51.923551 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:51.923910 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:51.924028 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:51.926901 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:51.926908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:51.926921 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:51.926947 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:51.928602 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:51.929170 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:51.929175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:51.929176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:51.929277 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:51.929658 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:51.929661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:51.929662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:51.930027 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:51.930132 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:51.930700 (+ 568us) server_negotiation.cc:299] Negotiation successful
0805 23:09:51.930972 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":432,"thread_start_us":161,"threads_started":1}
I20260805 23:09:51.932035 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:51.917855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43225 (local address 127.0.0.1:48264)
0805 23:09:51.918347 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:51.918402 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:51.919612 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:51.920359 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:51.920365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:51.920711 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:51.920965 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:51.920977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:51.922099 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:51.922106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:51.923411 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:51.923426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:51.924085 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:51.924093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:51.924286 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:51.924946 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:51.924971 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:51.926734 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:51.928733 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:51.928742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:51.928750 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:51.929067 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:51.929359 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:51.929361 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:51.929362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:51.929475 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:51.930156 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:51.930161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:51.930411 (+ 250us) client_negotiation.cc:769] Sending connection context
0805 23:09:51.930619 (+ 208us) client_negotiation.cc:240] Negotiation successful
0805 23:09:51.931039 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":131,"thread_start_us":139,"threads_started":1}
I20260805 23:09:51.934809 3107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:51.935669 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:51.935347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34028)
0805 23:09:51.935518 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:51.935597 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260805 23:09:51.935906 3031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
I20260805 23:09:51.936937 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:51.936699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34040)
0805 23:09:51.936820 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:51.936876 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260805 23:09:51.937093 3107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:43203: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab.
Entry for principal kudu/127.3.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab.
I20260805 23:09:51.949704 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.125:43203
--webserver_interface=127.3.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab
--principal=kudu/127.3.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:43225,127.3.12.125:43203,127.3.12.124:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:09:51.986338 3107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } attempt: 1
I20260805 23:09:51.987226 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:51.986914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34052)
0805 23:09:51.987074 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:51.987147 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260805 23:09:51.988023 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:51.987810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34068)
0805 23:09:51.987914 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:51.987965 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260805 23:09:51.988204 3107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:43203: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
I20260805 23:09:52.056383 3107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } attempt: 2
I20260805 23:09:52.057229 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.056905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34070)
0805 23:09:52.057084 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.057154 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260805 23:09:52.058177 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.057905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34084)
0805 23:09:52.058057 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.058117 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260805 23:09:52.058378 3107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:43203: Network error: Client connection negotiation failed: client connection to 127.3.12.125:43203: connect: Connection refused (error 111)
W20260805 23:09:52.064965 3117 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 23:09:52.065234 3117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:52.065305 3117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:52.065404 3117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:52.069526 3117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:52.069585 3117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:52.069607 3117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:52.069643 3117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:52.069677 3117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:52.074967 3117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:43225,127.3.12.125:43203,127.3.12.124:33789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab
--principal=kudu/127.3.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.125:43203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:52.076004 3117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:52.077123 3117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:52.085754 3134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:52.085779 3131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:52.085990 3132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:52.085811 3117 server_base.cc:1034] running on GCE node
I20260805 23:09:52.086612 3117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:52.087752 3117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:52.088908 3117 hybrid_clock.cc:648] HybridClock initialized: now 1785971392088888 us; error 33 us; skew 500 ppm
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:52.096458 3117 init.cc:377] Logged in from keytab as kudu/127.3.12.125@KRBTEST.COM (short username kudu)
I20260805 23:09:52.097404 3117 webserver.cc:460] Webserver started at http://127.3.12.125:35993/ using document root <none> and password file <none>
I20260805 23:09:52.097970 3117 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:52.098033 3117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:52.098248 3117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:52.100389 3117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/data/instance:
uuid: "59aa1905dacb425f9d76d1226f44179d"
format_stamp: "Formatted at 2026-08-05 23:09:52 on dist-test-slave-2d19"
server_key: "e5fe343158decb9ccc502419816f8b2e"
server_key_iv: "d7fb77d59bfead44dd15f3b006c74727"
server_key_version: "encryptionkey@0"
I20260805 23:09:52.100967 3117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/wal/instance:
uuid: "59aa1905dacb425f9d76d1226f44179d"
format_stamp: "Formatted at 2026-08-05 23:09:52 on dist-test-slave-2d19"
server_key: "e5fe343158decb9ccc502419816f8b2e"
server_key_iv: "d7fb77d59bfead44dd15f3b006c74727"
server_key_version: "encryptionkey@0"
I20260805 23:09:52.104746 3117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260805 23:09:52.107149 3148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:52.108546 3117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:52.108686 3117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/wal
uuid: "59aa1905dacb425f9d76d1226f44179d"
format_stamp: "Formatted at 2026-08-05 23:09:52 on dist-test-slave-2d19"
server_key: "e5fe343158decb9ccc502419816f8b2e"
server_key_iv: "d7fb77d59bfead44dd15f3b006c74727"
server_key_version: "encryptionkey@0"
I20260805 23:09:52.108798 3117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:52.122387 3117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:52.126137 3117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:52.126348 3117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:52.133599 3117 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.125:43203
I20260805 23:09:52.133920 3232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.125:43203 every 8 connection(s)
I20260805 23:09:52.134675 3117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
I20260805 23:09:52.137576 3234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:52.137770 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3117
I20260805 23:09:52.137933 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-1/wal/instance
I20260805 23:09:52.138250 3121 external_mini_cluster.cc:1428] Setting key cfd41e1b72f4e1b6e67a0e33ab45a104
I20260805 23:09:52.142087 3234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.12.124@KRBTEST.COM; defaulting to no policy
I20260805 23:09:52.149593 3107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.124@KRBTEST.COM" created.
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
I20260805 23:09:52.160517 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.144615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.3.12.126:43225)
0805 23:09:52.144808 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.144815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.145855 (+ 1040us) server_negotiation.cc:407] Connection header received
0805 23:09:52.147178 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.147181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.147239 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.147326 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.149005 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.149618 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.151128 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.151497 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.151618 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.154919 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.154924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.154929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.154965 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.157636 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.158352 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.158357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.158359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.158448 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.158801 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.158805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.158806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.159050 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.159148 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.160130 (+ 982us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.160371 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":63}
I20260805 23:09:52.160517 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.150000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34086)
0805 23:09:52.150175 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.150192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:52.150274 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:52.150869 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:52.150873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:52.151079 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:52.151196 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.151203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.152571 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.152574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.153383 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.153398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.154244 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.154248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.154359 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.154869 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:52.154880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:52.156207 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:52.158551 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.158558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.158563 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.158829 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.159154 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.159159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.159163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.159268 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.159892 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:52.159895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:52.159989 (+ 94us) client_negotiation.cc:769] Sending connection context
0805 23:09:52.160093 (+ 104us) client_negotiation.cc:240] Negotiation successful
0805 23:09:52.160315 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":5}
I20260805 23:09:52.161517 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.150135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.3.12.125:43203)
0805 23:09:52.150488 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.150497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.150524 (+ 27us) server_negotiation.cc:407] Connection header received
0805 23:09:52.150557 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.150561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.150710 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.150840 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.151288 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.152440 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.153529 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.154103 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.154599 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.156382 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.156391 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.156407 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.156436 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.158424 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.158945 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.158951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.158953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.159034 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.159368 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.159370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.159372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.159783 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.159923 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.160961 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.161138 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":252,"thread_start_us":173,"threads_started":1}
I20260805 23:09:52.161550 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.145055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43225 (local address 127.0.0.1:48280)
0805 23:09:52.145649 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.145711 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:52.146919 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:52.147556 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:52.147566 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:52.148065 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:52.148834 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.148858 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.149772 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.149778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.150960 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.150973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.151652 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.151655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.151800 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.152545 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:52.152567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:52.154464 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:52.157743 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.157756 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.157769 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.158234 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.158558 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.158561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.158563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.158700 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.159429 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:52.159439 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:52.159832 (+ 393us) client_negotiation.cc:769] Sending connection context
0805 23:09:52.160057 (+ 225us) client_negotiation.cc:240] Negotiation successful
0805 23:09:52.160454 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":7,"thread_start_us":130,"threads_started":1}
I20260805 23:09:52.163242 3234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:52.163486 3107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:52.164302 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.164004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40342)
0805 23:09:52.164126 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.164184 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260805 23:09:52.165092 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.164828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40354)
0805 23:09:52.164977 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.165031 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260805 23:09:52.165231 3107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:33789: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab.
Entry for principal kudu/127.3.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab.
I20260805 23:09:52.172458 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.163815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34102)
0805 23:09:52.163917 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.163931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:52.164002 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:52.164342 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:52.164347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:52.164527 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:52.164666 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.164673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.165647 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.165650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.166178 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.166186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.166637 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.166640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.166789 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.167187 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:52.167221 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:52.168658 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:52.170593 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.170596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.170597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.170814 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.171210 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.171213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.171214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.171304 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.171788 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:52.171794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:52.171932 (+ 138us) client_negotiation.cc:769] Sending connection context
0805 23:09:52.172116 (+ 184us) client_negotiation.cc:240] Negotiation successful
0805 23:09:52.172319 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260805 23:09:52.172662 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.163829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.3.12.125:43203)
0805 23:09:52.164143 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.164147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.164158 (+ 11us) server_negotiation.cc:407] Connection header received
0805 23:09:52.164187 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.164190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.164226 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.164322 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.164792 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.165537 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.166288 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.166536 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.166638 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.168812 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.168814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.168816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.168866 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.170453 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.171002 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.171005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.171006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.171083 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.171445 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.171449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.171451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.171660 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.171788 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.172191 (+ 403us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.172426 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":259}
I20260805 23:09:52.172897 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.124:33789
--webserver_interface=127.3.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab
--principal=kudu/127.3.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:43225,127.3.12.125:43203,127.3.12.124:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:09:52.173595 3234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:52.174325 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.174056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40356)
0805 23:09:52.174174 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.174268 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 23:09:52.175052 3150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
I20260805 23:09:52.176083 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.175843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40372)
0805 23:09:52.175978 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.176029 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260805 23:09:52.176360 3234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:33789: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
I20260805 23:09:52.196408 3107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } attempt: 1
I20260805 23:09:52.197320 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.196958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40384)
0805 23:09:52.197138 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.197217 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260805 23:09:52.198153 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.197908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40388)
0805 23:09:52.198024 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.198080 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260805 23:09:52.198393 3107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:33789: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
I20260805 23:09:52.225660 3234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } attempt: 1
I20260805 23:09:52.226620 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.226288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40402)
0805 23:09:52.226461 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.226535 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260805 23:09:52.227423 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.227214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40404)
0805 23:09:52.227317 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.227365 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260805 23:09:52.227576 3234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:33789: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
I20260805 23:09:52.273597 3107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } attempt: 2
I20260805 23:09:52.274530 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.274146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40414)
0805 23:09:52.274336 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.274417 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260805 23:09:52.275394 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.275134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40428)
0805 23:09:52.275260 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.275319 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260805 23:09:52.275586 3107 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:33789: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
W20260805 23:09:52.293845 3244 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 23:09:52.294206 3244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:52.294271 3244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:52.294348 3244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260805 23:09:52.295760 3234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } attempt: 2
I20260805 23:09:52.296715 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.296397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40436)
0805 23:09:52.296565 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.296632 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260805 23:09:52.297499 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.297268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40438)
0805 23:09:52.297384 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.297440 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260805 23:09:52.297657 3234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:33789: Network error: Client connection negotiation failed: client connection to 127.3.12.124:33789: connect: Connection refused (error 111)
W20260805 23:09:52.298810 3244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:52.298883 3244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:52.298926 3244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:52.298975 3244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:52.299019 3244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:52.305188 3244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:43225,127.3.12.125:43203,127.3.12.124:33789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab
--principal=kudu/127.3.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.124:33789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:52.306313 3244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:52.307387 3244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:52.316407 3253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:52.316437 3251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:52.316407 3250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:52.316653 3244 server_base.cc:1034] running on GCE node
I20260805 23:09:52.317369 3244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:52.318554 3244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:52.319765 3244 hybrid_clock.cc:648] HybridClock initialized: now 1785971392319748 us; error 38 us; skew 500 ppm
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:52.322738 3244 init.cc:377] Logged in from keytab as kudu/127.3.12.124@KRBTEST.COM (short username kudu)
I20260805 23:09:52.323624 3244 webserver.cc:460] Webserver started at http://127.3.12.124:43757/ using document root <none> and password file <none>
I20260805 23:09:52.324198 3244 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:52.324331 3244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:52.324568 3244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:52.326712 3244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/data/instance:
uuid: "b51e375671954344ada9887a3248ff39"
format_stamp: "Formatted at 2026-08-05 23:09:52 on dist-test-slave-2d19"
server_key: "ebcf5b6532f81315f41c239779835936"
server_key_iv: "6491974f4aafd698c2d5179cde22075e"
server_key_version: "encryptionkey@0"
I20260805 23:09:52.327299 3244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/wal/instance:
uuid: "b51e375671954344ada9887a3248ff39"
format_stamp: "Formatted at 2026-08-05 23:09:52 on dist-test-slave-2d19"
server_key: "ebcf5b6532f81315f41c239779835936"
server_key_iv: "6491974f4aafd698c2d5179cde22075e"
server_key_version: "encryptionkey@0"
I20260805 23:09:52.331212 3244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260805 23:09:52.333842 3267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:52.335176 3244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 23:09:52.335322 3244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/wal
uuid: "b51e375671954344ada9887a3248ff39"
format_stamp: "Formatted at 2026-08-05 23:09:52 on dist-test-slave-2d19"
server_key: "ebcf5b6532f81315f41c239779835936"
server_key_iv: "6491974f4aafd698c2d5179cde22075e"
server_key_version: "encryptionkey@0"
I20260805 23:09:52.335438 3244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:52.355414 3244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:52.358716 3244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:52.358932 3244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:52.366237 3244 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.124:33789
I20260805 23:09:52.366551 3341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.124:33789 every 8 connection(s)
I20260805 23:09:52.367280 3244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
I20260805 23:09:52.370177 3342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:52.371440 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3244
I20260805 23:09:52.371611 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785971095475883-3121-0/minicluster-data/master-2/wal/instance
I20260805 23:09:52.371919 3121 external_mini_cluster.cc:1428] Setting key c1e5714f18d2393fde3609bd53a9731c
I20260805 23:09:52.374790 3107 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } attempt: 3
I20260805 23:09:52.374727 3342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:52.388219 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.375327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40448)
0805 23:09:52.375463 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.375479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:52.375584 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:52.375959 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:52.375963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:52.376209 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:52.376374 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.376383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.378323 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.378327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.379163 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.379175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.380042 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.380046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.380127 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.380729 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:52.380743 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:52.382392 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:52.386128 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.386131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.386135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.386769 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.387094 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.387101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.387105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.387205 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.387703 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:52.387707 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:52.387831 (+ 124us) client_negotiation.cc:769] Sending connection context
0805 23:09:52.387974 (+ 143us) client_negotiation.cc:240] Negotiation successful
0805 23:09:52.388090 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":2}
I20260805 23:09:52.388842 3234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } attempt: 3
I20260805 23:09:52.389109 3346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.373496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.3.12.124:33789)
0805 23:09:52.373926 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.373947 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.373992 (+ 45us) server_negotiation.cc:407] Connection header received
0805 23:09:52.374837 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.374862 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.375305 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.375659 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.376675 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.378105 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.379332 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.379912 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.380445 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.383234 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.383240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.383243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.383279 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.385762 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.386261 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.386267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.386274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.386389 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.386677 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.386682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.386683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.387086 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.387382 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.387684 (+ 302us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.388011 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":277,"thread_start_us":114,"threads_started":1}
I20260805 23:09:52.389202 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.375374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.3.12.124:33789)
0805 23:09:52.375668 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.375677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.375692 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:09:52.375732 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.375737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.375789 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.376519 (+ 730us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.376659 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.378159 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.379320 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.379909 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.380084 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.382551 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.382557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.382574 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.382609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.385989 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.386874 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.386878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.386879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.386977 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.387342 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.387346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.387347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.387583 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.388770 (+ 1187us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.388876 (+ 106us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.388987 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":293,"server-negotiator.queue_time_us":229,"spinlock_wait_cycles":4736,"thread_start_us":116,"threads_started":1}
I20260805 23:09:52.389420 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.377006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.3.12.126:43225)
0805 23:09:52.377138 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.377142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.377491 (+ 349us) server_negotiation.cc:407] Connection header received
0805 23:09:52.377679 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.377682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.377736 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.377779 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.378648 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.379131 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.380312 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.380568 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.380688 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.385085 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.385089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.385091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.385112 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.386617 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.387179 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.387184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.387188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.387290 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.387633 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.387637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.387639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.388633 (+ 994us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.388714 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.389147 (+ 433us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.389266 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":754,"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":9728}
I20260805 23:09:52.389871 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.376948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:43225 (local address 127.0.0.1:48288)
0805 23:09:52.377366 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.377416 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:52.377573 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:52.377954 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:52.377959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:52.378416 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:52.378533 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.378538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.379241 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.379246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.380180 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.380189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.380738 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.380744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.382105 (+ 1361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.382985 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:52.383002 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:52.384877 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:52.386726 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.386737 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.386743 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.387075 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.387404 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.387407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.387409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.387521 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.388742 (+ 1221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:52.388748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:52.388878 (+ 130us) client_negotiation.cc:769] Sending connection context
0805 23:09:52.389542 (+ 664us) client_negotiation.cc:240] Negotiation successful
0805 23:09:52.389717 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"spinlock_wait_cycles":1408,"thread_start_us":150,"threads_started":1}
I20260805 23:09:52.391525 3342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:52.393858 3107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2: Bootstrap starting.
I20260805 23:09:52.396771 3107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:52.397821 3107 log.cc:824] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
I20260805 23:09:52.400866 3107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2: No bootstrap required, opened a new log
I20260805 23:09:52.403622 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.391953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.3.12.126:43225)
0805 23:09:52.392083 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.392087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.392101 (+ 14us) server_negotiation.cc:407] Connection header received
0805 23:09:52.392201 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.392204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.392245 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.392339 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.393191 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.393910 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.395066 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.395377 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.395523 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.399402 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.399406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.399409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.399435 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.401224 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.401726 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.401731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.401733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.401819 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.402146 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.402149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.402151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.402379 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.402661 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.403337 (+ 676us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.403462 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260805 23:09:52.404031 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.392585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.3.12.125:43203)
0805 23:09:52.392689 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.392693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.393320 (+ 627us) server_negotiation.cc:407] Connection header received
0805 23:09:52.393494 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.393497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.393540 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.393597 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.394147 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.394905 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.396313 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.396664 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.396785 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.400200 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.400204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.400208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.400237 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.401960 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.402515 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.402519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.402521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.402609 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.402943 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.402948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.402950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.403289 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.403377 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.403750 (+ 373us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.403865 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260805 23:09:52.404630 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.389301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40458)
0805 23:09:52.390395 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.390408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:52.390491 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:52.391455 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:52.391458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:52.391674 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:52.391792 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.391800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.393442 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.393445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.395058 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.395065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.395784 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.395788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.395865 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.396994 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:52.397009 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:52.399656 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:52.402933 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.402936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.402939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.403217 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.403568 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.403572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.403574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.403670 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.404103 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:52.404106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:52.404196 (+ 90us) client_negotiation.cc:769] Sending connection context
0805 23:09:52.404331 (+ 135us) client_negotiation.cc:240] Negotiation successful
0805 23:09:52.404459 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":836,"spinlock_wait_cycles":4480}
I20260805 23:09:52.405246 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.392519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:43203 (local address 127.0.0.1:34118)
0805 23:09:52.393256 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.393269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:52.393387 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:52.393726 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:52.393729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:52.393887 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:52.394020 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.394029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.395206 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.395209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.396118 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.396131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.396809 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.396814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.396947 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.397464 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:52.397479 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:52.400051 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:52.402076 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.402082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.402086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.402379 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.402730 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.402733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.402734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.402825 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.403443 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:52.403446 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:52.403535 (+ 89us) client_negotiation.cc:769] Sending connection context
0805 23:09:52.404856 (+ 1321us) client_negotiation.cc:240] Negotiation successful
0805 23:09:52.405041 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":92}
I20260805 23:09:52.405365 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.389562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.3.12.124:33789)
0805 23:09:52.389757 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.389760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.391233 (+ 1473us) server_negotiation.cc:407] Connection header received
0805 23:09:52.391274 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.391277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.391317 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.392156 (+ 839us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.392240 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.393300 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.395186 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.395634 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.396075 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.399773 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.399778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.399782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.399812 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.402798 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.403352 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.403355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.403357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.403440 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.403791 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.403794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.403796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.403988 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.404803 (+ 815us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.404880 (+ 77us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.405132 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":116}
I20260805 23:09:52.406286 3342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:52.406344 3107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } }
I20260805 23:09:52.406558 3107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9496447aaa8b49e18e2dc6c2f7ea70a2, State: Initialized, Role: FOLLOWER
I20260805 23:09:52.406970 3107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } }
I20260805 23:09:52.407473 3351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.407603 3351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:52.408346 3107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:52.408699 3234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d: Bootstrap starting.
I20260805 23:09:52.412381 3234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d: Neither blocks nor log segments found. Creating new log.
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971392, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
W20260805 23:09:52.413892 3370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:52.414004 3370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:52.414059 3234 log.cc:824] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:09:52 dist-test-slave-2d19 krb5kdc[2989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
I20260805 23:09:52.417513 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.406737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:33789 (local address 127.0.0.1:40474)
0805 23:09:52.408473 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:52.408489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:52.408590 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:52.408916 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:52.408919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:52.409092 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:52.409202 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.409210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.410166 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.410169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.410936 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:52.410947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.411512 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.411515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:52.411583 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.412088 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:52.412106 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:52.413652 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:52.415784 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.415789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.415792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.416075 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.416397 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:52.416399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:52.416401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:52.416493 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:52.416971 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:52.416974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:52.417084 (+ 110us) client_negotiation.cc:769] Sending connection context
0805 23:09:52.417187 (+ 103us) client_negotiation.cc:240] Negotiation successful
0805 23:09:52.417341 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1630}
I20260805 23:09:52.413743 3009 master_runner.cc:186] Error getting master registration for 127.3.12.125:43203: OK, instance_id { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" instance_seqno: 1785971392131584 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 23:09:52.417932 3009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 23:09:52.418030 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.406932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.3.12.124:33789)
0805 23:09:52.407023 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.407026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.408538 (+ 1512us) server_negotiation.cc:407] Connection header received
0805 23:09:52.408681 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.408683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.408727 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.408771 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.409319 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.410045 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.411059 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.411325 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.411415 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.413808 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.413814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.413819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.413859 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.415655 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.416176 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.416181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.416186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.416291 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.416593 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.416596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.416598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.416832 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.416935 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.417715 (+ 780us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.417829 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260805 23:09:52.420076 3234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d: No bootstrap required, opened a new log
I20260805 23:09:52.421628 3342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39: Bootstrap starting.
I20260805 23:09:52.421826 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.405101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.3.12.125:43203)
0805 23:09:52.405202 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.405206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.405446 (+ 240us) server_negotiation.cc:407] Connection header received
0805 23:09:52.405631 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.405636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.405688 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.405743 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.406643 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.407341 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.408435 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.408740 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.408836 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.417641 (+ 8805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.417648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.417652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.417680 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.419359 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.419905 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.419909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.419910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.420002 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.420377 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.420380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.420382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.420636 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.420814 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.421558 (+ 744us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.421685 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260805 23:09:52.424568 3342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:52.424726 3234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } }
I20260805 23:09:52.424989 3234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59aa1905dacb425f9d76d1226f44179d, State: Initialized, Role: FOLLOWER
I20260805 23:09:52.425190 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:52.425361 3234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } }
I20260805 23:09:52.425540 3342 log.cc:824] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:52.426728 3071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48294
I20260805 23:09:52.426913 3234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:52.426990 3071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 23:09:52.427125 3071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:52.428125 3342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39: No bootstrap required, opened a new log
I20260805 23:09:52.428579 3071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } }
I20260805 23:09:52.429616 3071 leader_election.cc:288] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [CANDIDATE]: Term 1 election: Requested vote from peers 59aa1905dacb425f9d76d1226f44179d (127.3.12.125:43203), b51e375671954344ada9887a3248ff39 (127.3.12.124:33789)
I20260805 23:09:52.430296 3197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "59aa1905dacb425f9d76d1226f44179d"
I20260805 23:09:52.432732 3197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:52.432526 3342 raft_consensus.cc:348] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } }
I20260805 23:09:52.432963 3342 raft_consensus.cc:729] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51e375671954344ada9887a3248ff39, State: Initialized, Role: FOLLOWER
I20260805 23:09:52.433312 3342 consensus_queue.cc:260] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } }
I20260805 23:09:52.434201 3197 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9496447aaa8b49e18e2dc6c2f7ea70a2 in term 1.
I20260805 23:09:52.434682 3031 leader_election.cc:302] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59aa1905dacb425f9d76d1226f44179d, 9496447aaa8b49e18e2dc6c2f7ea70a2; no voters:
I20260805 23:09:52.434780 3342 sys_catalog.cc:564] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:52.435024 3351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:52.435292 3351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [term 1 LEADER]: Becoming Leader. State: Replica: 9496447aaa8b49e18e2dc6c2f7ea70a2, State: Running, Role: LEADER
I20260805 23:09:52.435690 3351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } }
I20260805 23:09:52.436501 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.436717 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:52.437865 3351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9496447aaa8b49e18e2dc6c2f7ea70a2. Latest consensus state: current_term: 1 leader_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.438016 3351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:52.438300 3307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b51e375671954344ada9887a3248ff39"
I20260805 23:09:52.438540 3307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:52.439994 3307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9496447aaa8b49e18e2dc6c2f7ea70a2 in term 1.
W20260805 23:09:52.441226 3413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:52.441315 3413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:52.441634 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.441785 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:52.442715 3417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:52.443562 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.432686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.3.12.124:33789)
0805 23:09:52.432823 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.432829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.433374 (+ 545us) server_negotiation.cc:407] Connection header received
0805 23:09:52.433577 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.433581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.433638 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.433704 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.434814 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.435647 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.436891 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.437242 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.437333 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.439446 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.439449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.439452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.439482 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.441414 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.441961 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.441965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.441967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.442061 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.442514 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.442519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.442520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.442764 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.442930 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.443282 (+ 352us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.443404 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260805 23:09:52.443825 3417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260805 23:09:52.452721 3389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260805 23:09:52.453896 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.433027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.3.12.126:43225)
0805 23:09:52.433151 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.433158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.443651 (+ 10493us) server_negotiation.cc:407] Connection header received
0805 23:09:52.443841 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.443848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.443913 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.443981 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.444962 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.445704 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.446801 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.447124 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.447201 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.449746 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.449750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.449752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.449779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.451602 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.452155 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.452159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.452161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.452286 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.452617 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.452621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.452623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.452857 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.453005 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.453556 (+ 551us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.453691 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260805 23:09:52.456614 3389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:52.462891 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:52.453167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.3.12.125:43203)
0805 23:09:52.453278 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.453282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.453294 (+ 12us) server_negotiation.cc:407] Connection header received
0805 23:09:52.453329 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.453331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.453374 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.453467 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.454704 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.455305 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.456309 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:52.456626 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:52.456731 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:52.458723 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:52.458727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:52.458729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:52.458758 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:52.460551 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.461078 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.461082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.461084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.461175 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:52.461484 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:52.461487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:52.461489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:52.461714 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:52.461874 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:52.462593 (+ 719us) server_negotiation.cc:299] Negotiation successful
0805 23:09:52.462719 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260805 23:09:52.464570 3307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [term 1 FOLLOWER]: Refusing update from remote peer 9496447aaa8b49e18e2dc6c2f7ea70a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:52.464802 3197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [term 1 FOLLOWER]: Refusing update from remote peer 9496447aaa8b49e18e2dc6c2f7ea70a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:52.465387 3401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:52.465623 3351 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:52.470196 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9496447aaa8b49e18e2dc6c2f7ea70a2. Latest consensus state: current_term: 1 leader_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.470263 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9496447aaa8b49e18e2dc6c2f7ea70a2. Latest consensus state: current_term: 1 leader_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.470376 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:52.470393 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:52.478716 3351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.478871 3351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:52.478995 3419 mvcc.cc:205] Tried to move back new op lower bound from 7315338823499632640 to 7315338823421079552. Current Snapshot: MvccSnapshot[applied={T|T < 7315338823499632640}]
I20260805 23:09:52.480396 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.480547 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:52.481207 3422 mvcc.cc:205] Tried to move back new op lower bound from 7315338823499632640 to 7315338823421079552. Current Snapshot: MvccSnapshot[applied={T|T < 7315338823499632640}]
I20260805 23:09:52.485383 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.485572 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2 [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:52.486683 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9496447aaa8b49e18e2dc6c2f7ea70a2" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 43225 } } peers { permanent_uuid: "59aa1905dacb425f9d76d1226f44179d" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 43203 } } peers { permanent_uuid: "b51e375671954344ada9887a3248ff39" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 33789 } } }
I20260805 23:09:52.486812 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:52.487185 3423 mvcc.cc:205] Tried to move back new op lower bound from 7315338823499632640 to 7315338823421079552. Current Snapshot: MvccSnapshot[applied={T|T < 7315338823499632640}]
I20260805 23:09:52.488718 3417 catalog_manager.cc:1347] Generated new cluster ID: 9c35887fdfff41c0a3ff1bd0a9927be6
I20260805 23:09:52.488790 3417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:52.517927 3417 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:52.521759 3417 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:52.535318 3417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9496447aaa8b49e18e2dc6c2f7ea70a2: Generated new TSK 0
I20260805 23:09:52.536319 3417 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 23:09:52.574293 3112 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.571599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.3.12.126:43225)
0805 23:09:52.571736 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.571741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.573160 (+ 1419us) server_negotiation.cc:407] Connection header received
0805 23:09:52.573371 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.573374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.573439 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.573494 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.574158 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48310: BlockingRecv error: recv got EOF from 127.0.0.1:48310 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260805 23:09:52.575744 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.572529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.3.12.125:43203)
0805 23:09:52.572643 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.572648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.574804 (+ 2156us) server_negotiation.cc:407] Connection header received
0805 23:09:52.574986 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.574990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.575040 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.575093 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.575608 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34144: BlockingRecv error: recv got EOF from 127.0.0.1:34144 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260805 23:09:52.577653 3347 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.576232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.3.12.124:33789)
0805 23:09:52.576439 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.576445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.576764 (+ 319us) server_negotiation.cc:407] Connection header received
0805 23:09:52.576935 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.576939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.576987 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.577039 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.577535 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40490: BlockingRecv error: recv got EOF from 127.0.0.1:40490 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260805 23:09:52.582365 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.580632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.3.12.125:43203)
0805 23:09:52.580760 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.580764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.581251 (+ 487us) server_negotiation.cc:407] Connection header received
0805 23:09:52.581478 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.581481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.581556 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.581610 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.582264 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34158: BlockingRecv error: recv got EOF from 127.0.0.1:34158 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260805 23:09:52.587344 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.585901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.3.12.125:43203)
0805 23:09:52.585999 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.586002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.586493 (+ 491us) server_negotiation.cc:407] Connection header received
0805 23:09:52.586662 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.586666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.586710 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.586759 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.587242 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34168: BlockingRecv error: recv got EOF from 127.0.0.1:34168 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260805 23:09:52.593081 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.591490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.3.12.125:43203)
0805 23:09:52.591591 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.591595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.592093 (+ 498us) server_negotiation.cc:407] Connection header received
0805 23:09:52.592372 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.592375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.592439 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.592494 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.592986 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34182: BlockingRecv error: recv got EOF from 127.0.0.1:34182 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260805 23:09:52.601017 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.599375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.3.12.125:43203)
0805 23:09:52.599527 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.599531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.600055 (+ 524us) server_negotiation.cc:407] Connection header received
0805 23:09:52.600273 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.600277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.600327 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.600380 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.600909 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34184: BlockingRecv error: recv got EOF from 127.0.0.1:34184 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260805 23:09:52.613499 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.611544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.3.12.125:43203)
0805 23:09:52.611667 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.611672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.612333 (+ 661us) server_negotiation.cc:407] Connection header received
0805 23:09:52.612601 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.612604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.612674 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.612731 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.613343 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34196: BlockingRecv error: recv got EOF from 127.0.0.1:34196 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260805 23:09:52.633978 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.632292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.3.12.125:43203)
0805 23:09:52.632425 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.632430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.633009 (+ 579us) server_negotiation.cc:407] Connection header received
0805 23:09:52.633207 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.633211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.633262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.633314 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.633860 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34202: BlockingRecv error: recv got EOF from 127.0.0.1:34202 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260805 23:09:52.669950 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.668637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.3.12.125:43203)
0805 23:09:52.668778 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.668783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.668982 (+ 199us) server_negotiation.cc:407] Connection header received
0805 23:09:52.669176 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.669179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.669230 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.669320 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.669814 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34206: BlockingRecv error: recv got EOF from 127.0.0.1:34206 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260805 23:09:52.737830 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.736455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.3.12.125:43203)
0805 23:09:52.736605 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.736612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.736818 (+ 206us) server_negotiation.cc:407] Connection header received
0805 23:09:52.737000 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.737003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.737124 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.737209 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.737712 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34222: BlockingRecv error: recv got EOF from 127.0.0.1:34222 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260805 23:09:52.869455 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:52.868331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.3.12.125:43203)
0805 23:09:52.868464 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:52.868467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:52.868590 (+ 123us) server_negotiation.cc:407] Connection header received
0805 23:09:52.868697 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:52.868700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:52.868756 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:52.868861 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:52.869332 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34230: BlockingRecv error: recv got EOF from 127.0.0.1:34230 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260805 23:09:53.128986 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:53.127829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.3.12.125:43203)
0805 23:09:53.127958 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:53.127962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:53.128064 (+ 102us) server_negotiation.cc:407] Connection header received
0805 23:09:53.128196 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:53.128198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:53.128267 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:53.128347 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:53.128866 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34242: BlockingRecv error: recv got EOF from 127.0.0.1:34242 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260805 23:09:53.442710 3413 catalog_manager.cc:1259] Loaded cluster ID: 9c35887fdfff41c0a3ff1bd0a9927be6
I20260805 23:09:53.442799 3413 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39: loading cluster ID for follower catalog manager: success
I20260805 23:09:53.444620 3413 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39: acquiring CA information for follower catalog manager: success
I20260805 23:09:53.445797 3413 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b51e375671954344ada9887a3248ff39: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 23:09:53.458061 3389 catalog_manager.cc:1259] Loaded cluster ID: 9c35887fdfff41c0a3ff1bd0a9927be6
I20260805 23:09:53.458151 3389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d: loading cluster ID for follower catalog manager: success
I20260805 23:09:53.459959 3389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d: acquiring CA information for follower catalog manager: success
I20260805 23:09:53.461282 3389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59aa1905dacb425f9d76d1226f44179d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 23:09:53.646934 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:53.643183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.3.12.125:43203)
0805 23:09:53.643414 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:53.643419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:53.643438 (+ 19us) server_negotiation.cc:407] Connection header received
0805 23:09:53.643560 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:53.643563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:53.643689 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:53.643802 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:53.644079 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.644645 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.645561 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.645785 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.645862 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:53.646175 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:53.646609 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:53.646704 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:53.646760 (+ 56us) server_negotiation.cc:299] Negotiation successful
0805 23:09:53.646807 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260805 23:09:53.653723 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:53.650127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.3.12.124:33789)
0805 23:09:53.650375 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:53.650379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:53.650392 (+ 13us) server_negotiation.cc:407] Connection header received
0805 23:09:53.650478 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:53.650481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:53.650603 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:53.650708 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:53.650928 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.651439 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.652312 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.652547 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.652634 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:53.652891 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:53.653320 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:53.653424 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:53.653472 (+ 48us) server_negotiation.cc:299] Negotiation successful
0805 23:09:53.653520 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260805 23:09:53.660802 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:53.656594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.3.12.125:43203)
0805 23:09:53.656729 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:53.656733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:53.657143 (+ 410us) server_negotiation.cc:407] Connection header received
0805 23:09:53.657260 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:53.657263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:53.657309 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:53.657382 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:53.657690 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.658252 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.659462 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.659732 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.659838 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:53.660285 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:53.660368 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:53.660443 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:53.660578 (+ 135us) server_negotiation.cc:299] Negotiation successful
0805 23:09:53.660637 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260805 23:09:53.661585 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:53.656594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.3.12.124:33789)
0805 23:09:53.656722 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:53.656726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:53.657279 (+ 553us) server_negotiation.cc:407] Connection header received
0805 23:09:53.657392 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:53.657395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:53.657444 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:53.657520 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:53.657849 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.658331 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.660315 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.660664 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.660766 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:53.661051 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:53.661133 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:53.661183 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:53.661360 (+ 177us) server_negotiation.cc:299] Negotiation successful
0805 23:09:53.661430 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260805 23:09:53.662297 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:53.656706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.3.12.126:43225)
0805 23:09:53.656950 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:53.656957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:53.656974 (+ 17us) server_negotiation.cc:407] Connection header received
0805 23:09:53.657100 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:53.657105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:53.657271 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:53.657416 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:53.658509 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.659297 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.660448 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:53.660928 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:53.661026 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:53.661442 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:53.661748 (+ 306us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:53.661844 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:53.661956 (+ 112us) server_negotiation.cc:299] Negotiation successful
0805 23:09:53.662017 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260805 23:09:53.665107 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3009
I20260805 23:09:53.689417 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3117
I20260805 23:09:53.702682 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3244
2026-08-05T23:09:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3375 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 05 23:09:53 dist-test-slave-2d19 krb5kdc[3536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 05 23:09:53 dist-test-slave-2d19 krb5kdc[3536](info): set up 2 sockets
Aug 05 23:09:53 dist-test-slave-2d19 krb5kdc[3536](info): commencing operation
krb5kdc: starting...
W20260805 23:09:54.912966 3121 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/test-user.keytab.
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-05T23:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-05T23:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
Entry for principal kudu/127.3.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab.
I20260805 23:09:55.048401 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40001
--webserver_interface=127.3.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:40001,127.3.12.125:34177,127.3.12.124:39325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260805 23:09:55.161062 3554 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 23:09:55.161427 3554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:55.161501 3554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:55.161577 3554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:55.165596 3554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:55.165685 3554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:55.165723 3554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:55.165761 3554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:55.165796 3554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:55.171058 3554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:40001,127.3.12.125:34177,127.3.12.124:39325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.126.keytab
--principal=kudu/127.3.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.126:40001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:55.172226 3554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:55.173331 3554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:55.182025 3559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:55.182036 3560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:55.182354 3554 server_base.cc:1034] running on GCE node
W20260805 23:09:55.182037 3562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:55.183059 3554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:55.184216 3554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:55.185420 3554 hybrid_clock.cc:648] HybridClock initialized: now 1785971395185405 us; error 30 us; skew 500 ppm
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:55.188465 3554 init.cc:377] Logged in from keytab as kudu/127.3.12.126@KRBTEST.COM (short username kudu)
I20260805 23:09:55.189405 3554 webserver.cc:460] Webserver started at http://127.3.12.126:43895/ using document root <none> and password file <none>
I20260805 23:09:55.189963 3554 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:55.190026 3554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:55.190251 3554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:55.192479 3554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/data/instance:
uuid: "73fbd5b2d1a84595a53023a105b20bef"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "625d7e8513764638708a7779ce872e62"
server_key_iv: "8d1ecad4b4d61237d53dbd7749255f6d"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.193058 3554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/wal/instance:
uuid: "73fbd5b2d1a84595a53023a105b20bef"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "625d7e8513764638708a7779ce872e62"
server_key_iv: "8d1ecad4b4d61237d53dbd7749255f6d"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.196830 3554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260805 23:09:55.199321 3574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:55.200762 3554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260805 23:09:55.200902 3554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/wal
uuid: "73fbd5b2d1a84595a53023a105b20bef"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "625d7e8513764638708a7779ce872e62"
server_key_iv: "8d1ecad4b4d61237d53dbd7749255f6d"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.201012 3554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:55.211232 3554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:55.214356 3554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:55.214566 3554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:55.222043 3554 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.126:40001
I20260805 23:09:55.222370 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.126:40001 every 8 connection(s)
I20260805 23:09:55.223033 3554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/data/info.pb
I20260805 23:09:55.225660 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3554
I20260805 23:09:55.225816 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-0/wal/instance
I20260805 23:09:55.226200 3121 external_mini_cluster.cc:1428] Setting key 487754af395c6c125aa05d53e4ad0448
I20260805 23:09:55.226526 3642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:55.232086 3642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.125@KRBTEST.COM" created.
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:55.250741 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.235521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40001 (local address 127.0.0.1:51684)
0805 23:09:55.236104 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.236162 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.237275 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.238158 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.238163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.238600 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.238898 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.238912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.240219 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.240222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.241186 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.241197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.242122 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.242128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.242303 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.242975 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.243004 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.244967 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.247508 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.247520 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.247533 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.247796 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.248223 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.248225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.248226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.248402 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.249276 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.249281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.249421 (+ 140us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.249587 (+ 166us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.249871 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":33,"thread_start_us":152,"threads_started":1}
I20260805 23:09:55.250746 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.236578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.3.12.126:40001)
0805 23:09:55.236963 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.236976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.237024 (+ 48us) server_negotiation.cc:407] Connection header received
0805 23:09:55.237618 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.237628 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.237912 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.238135 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.239079 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.240069 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.241315 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.241853 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.242085 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.245154 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.245162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.245180 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.245211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.247329 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.247939 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.247946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.247948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.248115 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.248521 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.248527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.248531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.249147 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.249312 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.249736 (+ 424us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.249960 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":262,"thread_start_us":93,"threads_started":1}
I20260805 23:09:55.253652 3642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:55.254552 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.254290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48790)
0805 23:09:55.254405 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.254488 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260805 23:09:55.254789 3578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
I20260805 23:09:55.255736 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.255490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48806)
0805 23:09:55.255621 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.255680 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260805 23:09:55.255894 3642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:34177: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab.
Entry for principal kudu/127.3.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab.
I20260805 23:09:55.261883 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.125:34177
--webserver_interface=127.3.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab
--principal=kudu/127.3.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:40001,127.3.12.125:34177,127.3.12.124:39325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:09:55.305120 3642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } attempt: 1
I20260805 23:09:55.306071 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.305767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48810)
0805 23:09:55.305922 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.305993 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260805 23:09:55.306787 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.306574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48812)
0805 23:09:55.306680 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.306734 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260805 23:09:55.306912 3642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:34177: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
I20260805 23:09:55.375137 3642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } attempt: 2
I20260805 23:09:55.376049 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.375728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48826)
0805 23:09:55.375895 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.375971 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260805 23:09:55.376803 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.376589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48836)
0805 23:09:55.376699 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.376750 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260805 23:09:55.376979 3642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.125:34177: Network error: Client connection negotiation failed: client connection to 127.3.12.125:34177: connect: Connection refused (error 111)
W20260805 23:09:55.377631 3653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 23:09:55.377965 3653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:55.377996 3653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:55.378080 3653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:55.382251 3653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:55.382339 3653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:55.382377 3653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:55.382436 3653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:55.382474 3653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:55.387954 3653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:40001,127.3.12.125:34177,127.3.12.124:39325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.125.keytab
--principal=kudu/127.3.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.125:34177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:55.389112 3653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:55.390151 3653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:55.398928 3664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:55.398964 3662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:55.398945 3661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:55.399190 3653 server_base.cc:1034] running on GCE node
I20260805 23:09:55.399806 3653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:55.400969 3653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:55.402176 3653 hybrid_clock.cc:648] HybridClock initialized: now 1785971395402160 us; error 28 us; skew 500 ppm
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:55.405004 3653 init.cc:377] Logged in from keytab as kudu/127.3.12.125@KRBTEST.COM (short username kudu)
I20260805 23:09:55.405862 3653 webserver.cc:460] Webserver started at http://127.3.12.125:33983/ using document root <none> and password file <none>
I20260805 23:09:55.406412 3653 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:55.406502 3653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:55.406729 3653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:55.408879 3653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/data/instance:
uuid: "0621a8e09c0843559e110224b4c498a2"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "bc038ec8dcf43d799d194d330b2dfabb"
server_key_iv: "eb79b23ab94889b5468673a89a3e085c"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.409439 3653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/wal/instance:
uuid: "0621a8e09c0843559e110224b4c498a2"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "bc038ec8dcf43d799d194d330b2dfabb"
server_key_iv: "eb79b23ab94889b5468673a89a3e085c"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.413108 3653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260805 23:09:55.415452 3676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:55.416756 3653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 23:09:55.416898 3653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/wal
uuid: "0621a8e09c0843559e110224b4c498a2"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "bc038ec8dcf43d799d194d330b2dfabb"
server_key_iv: "eb79b23ab94889b5468673a89a3e085c"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.417006 3653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:55.450169 3653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:55.453393 3653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:55.453608 3653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:55.460943 3653 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.125:34177
I20260805 23:09:55.461262 3743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.125:34177 every 8 connection(s)
I20260805 23:09:55.461956 3653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/data/info.pb
I20260805 23:09:55.464870 3744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:55.468137 3642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } attempt: 3
I20260805 23:09:55.468994 3744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:55.469563 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3653
I20260805 23:09:55.469702 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-1/wal/instance
I20260805 23:09:55.470083 3121 external_mini_cluster.cc:1428] Setting key 9629a4e2f6de1753b73367192107d091
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.12.124@KRBTEST.COM; defaulting to no policy
I20260805 23:09:55.483381 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.468738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48846)
0805 23:09:55.468919 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.468933 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.469019 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.471860 (+ 2841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.471866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.472108 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.472302 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.472313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.474216 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.474219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.475083 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.475096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.476060 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.476064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.476302 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.477353 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.477369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.478787 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.481277 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.481282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.481285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.481551 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.481955 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.481958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.481960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.482050 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.482672 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.482675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.482784 (+ 109us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.482919 (+ 135us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.483078 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":61,"spinlock_wait_cycles":7936}
I20260805 23:09:55.483405 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.470910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.3.12.126:40001)
0805 23:09:55.471067 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.471072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.471446 (+ 374us) server_negotiation.cc:407] Connection header received
0805 23:09:55.471794 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.471797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.471856 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.471954 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.473168 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.473923 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.475366 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.476338 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.476450 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.479604 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.479608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.479610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.479633 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.481325 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.481831 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.481834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.481835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.481975 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.482321 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.482325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.482326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.482505 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.482616 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.483013 (+ 397us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.483218 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":1792}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.12.124@KRBTEST.COM" created.
I20260805 23:09:55.484383 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.469535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.3.12.125:34177)
0805 23:09:55.470014 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.470036 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.470083 (+ 47us) server_negotiation.cc:407] Connection header received
0805 23:09:55.471151 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.471178 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.471614 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.472470 (+ 856us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.472784 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.474109 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.475229 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.475920 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.476119 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.478974 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.478982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.478998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.479029 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.481138 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.481661 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.481670 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.481672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.481805 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.482150 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.482154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.482155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.482555 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.482704 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.483088 (+ 384us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.483299 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":134,"threads_started":1}
I20260805 23:09:55.484390 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.470901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40001 (local address 127.0.0.1:51698)
0805 23:09:55.471308 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.471356 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.471610 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.472017 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.472026 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.472604 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.472996 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.473014 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.474051 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.474054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.475253 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.475262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.476525 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.476533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.476661 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.477408 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.477461 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.479220 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.481426 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.481434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.481443 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.481710 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.482084 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.482088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.482089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.482214 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.482616 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.482624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.482765 (+ 141us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.482949 (+ 184us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.483420 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":115,"thread_start_us":81,"threads_started":1}
I20260805 23:09:55.486240 3744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:55.486462 3642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:55.487185 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.486924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43838)
0805 23:09:55.487087 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.487130 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260805 23:09:55.488041 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.487704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43848)
0805 23:09:55.487869 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.487964 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260805 23:09:55.488190 3642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:39325: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
I20260805 23:09:55.495330 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.486988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.3.12.125:34177)
0805 23:09:55.487394 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.487398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.487414 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:09:55.487449 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.487452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.487494 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.487580 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.488358 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.488851 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.489584 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.489835 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.489941 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.492213 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.492216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.492218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.492239 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.493641 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.494113 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.494115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.494116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.494185 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.494488 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.494491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.494492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.494655 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.494730 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.495083 (+ 353us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.495197 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":317}
I20260805 23:09:55.495352 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.486764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48856)
0805 23:09:55.486999 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.487014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.487106 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.487889 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.487893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.488089 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.488224 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.488231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.488960 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.488963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.489471 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.489478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.489937 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.489940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.490074 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.490547 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.490558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.492079 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.493759 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.493764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.493766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.494019 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.494278 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.494284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.494289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.494388 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.494812 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.494814 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.494897 (+ 83us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.495033 (+ 136us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.495144 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":20}
I20260805 23:09:55.496136 3744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:55.496811 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.496563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43850)
0805 23:09:55.496684 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.496763 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260805 23:09:55.497079 3681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
I20260805 23:09:55.497840 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.497659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43862)
0805 23:09:55.497751 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.497794 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260805 23:09:55.498023 3744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:39325: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab.
Entry for principal kudu/127.3.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab.
I20260805 23:09:55.503401 3121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
/tmp/dist-test-taskqJ6txj/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.124:39325
--webserver_interface=127.3.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab
--principal=kudu/127.3.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.12.84:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.12.126:40001,127.3.12.125:34177,127.3.12.124:39325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260805 23:09:55.519358 3642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } attempt: 1
I20260805 23:09:55.520301 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.519924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43868)
0805 23:09:55.520111 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.520186 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260805 23:09:55.521078 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.520833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43876)
0805 23:09:55.520948 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.521017 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260805 23:09:55.521229 3642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:39325: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
I20260805 23:09:55.547312 3744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } attempt: 1
I20260805 23:09:55.548206 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.547898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43888)
0805 23:09:55.548052 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.548127 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260805 23:09:55.549083 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.548860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43892)
0805 23:09:55.548968 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.549019 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260805 23:09:55.549207 3744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:39325: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
I20260805 23:09:55.596402 3642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } attempt: 2
I20260805 23:09:55.597301 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.597000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43908)
0805 23:09:55.597157 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.597222 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260805 23:09:55.598115 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.597920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43912)
0805 23:09:55.598015 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.598060 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260805 23:09:55.598318 3642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:39325: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
I20260805 23:09:55.617394 3744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } attempt: 2
I20260805 23:09:55.618299 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.617990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43916)
0805 23:09:55.618142 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.618217 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260805 23:09:55.619215 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.618977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43924)
0805 23:09:55.619103 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.619158 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260805 23:09:55.619400 3744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.12.124:39325: Network error: Client connection negotiation failed: client connection to 127.3.12.124:39325: connect: Connection refused (error 111)
W20260805 23:09:55.621624 3758 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260805 23:09:55.621896 3758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260805 23:09:55.621948 3758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260805 23:09:55.622016 3758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260805 23:09:55.626088 3758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260805 23:09:55.626140 3758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260805 23:09:55.626159 3758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260805 23:09:55.626178 3758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260805 23:09:55.626235 3758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260805 23:09:55.631654 3758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.12.84:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.12.126:40001,127.3.12.125:34177,127.3.12.124:39325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/krb5kdc/kudu_127.3.12.124.keytab
--principal=kudu/127.3.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.12.124:39325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 05 Aug 2026 22:32:10 UTC on ccycloud.quasar-zhglpi.root.comops.site
build id 4569239
I20260805 23:09:55.632753 3758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 23:09:55.633780 3758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 23:09:55.642325 3766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:55.642354 3764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 23:09:55.642336 3763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 23:09:55.642465 3758 server_base.cc:1034] running on GCE node
I20260805 23:09:55.643240 3758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260805 23:09:55.644400 3758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260805 23:09:55.645560 3758 hybrid_clock.cc:648] HybridClock initialized: now 1785971395645538 us; error 37 us; skew 500 ppm
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260805 23:09:55.648418 3758 init.cc:377] Logged in from keytab as kudu/127.3.12.124@KRBTEST.COM (short username kudu)
I20260805 23:09:55.649325 3758 webserver.cc:460] Webserver started at http://127.3.12.124:35865/ using document root <none> and password file <none>
I20260805 23:09:55.649847 3758 fs_manager.cc:359] Metadata directory not provided
I20260805 23:09:55.649948 3758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 23:09:55.650143 3758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 23:09:55.652127 3758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/data/instance:
uuid: "401f2557af7d4b58837b4750de85bbd6"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "b52dc00ded380082672abcd5281c365d"
server_key_iv: "1117baa1e6735e61dd16456f4756e723"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.652704 3758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/wal/instance:
uuid: "401f2557af7d4b58837b4750de85bbd6"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "b52dc00ded380082672abcd5281c365d"
server_key_iv: "1117baa1e6735e61dd16456f4756e723"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.656415 3758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260805 23:09:55.658772 3774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 23:09:55.660058 3758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 23:09:55.660181 3758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/data,/tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/wal
uuid: "401f2557af7d4b58837b4750de85bbd6"
format_stamp: "Formatted at 2026-08-05 23:09:55 on dist-test-slave-2d19"
server_key: "b52dc00ded380082672abcd5281c365d"
server_key_iv: "1117baa1e6735e61dd16456f4756e723"
server_key_version: "encryptionkey@0"
I20260805 23:09:55.660310 3758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 23:09:55.680820 3758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 23:09:55.684092 3758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 23:09:55.684247 3758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 23:09:55.691677 3758 rpc_server.cc:307] RPC server started. Bound to: 127.3.12.124:39325
I20260805 23:09:55.692034 3844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.12.124:39325 every 8 connection(s)
I20260805 23:09:55.692866 3758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/data/info.pb
I20260805 23:09:55.695989 3845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 23:09:55.697499 3642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } attempt: 3
I20260805 23:09:55.700897 3121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu as pid 3758
I20260805 23:09:55.701046 3121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqJ6txj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785971095475883-3121-0/minicluster-data/master-2/wal/instance
I20260805 23:09:55.701391 3121 external_mini_cluster.cc:1428] Setting key 9f07ea27c7122aa84d0096ff02361c77
I20260805 23:09:55.701853 3845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.126@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:55.710587 3744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } attempt: 3
I20260805 23:09:55.713651 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.698162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43926)
0805 23:09:55.698331 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.698348 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.698451 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.700742 (+ 2291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.700746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.700958 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.701083 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.701095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.703101 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.703105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.703854 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.703866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.705744 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.705748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.705849 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.706459 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.706474 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.708236 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.710958 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.710961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.710963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.711226 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.711619 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.711622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.711624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.711872 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.712811 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.712815 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.712928 (+ 113us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.713271 (+ 343us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.713438 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260805 23:09:55.715595 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.698804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.3.12.124:39325)
0805 23:09:55.699242 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.699258 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.699296 (+ 38us) server_negotiation.cc:407] Connection header received
0805 23:09:55.699982 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.699999 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.700414 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.700813 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.701362 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.702945 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.703998 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.705043 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.705256 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.708484 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.708492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.708508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.708548 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.710824 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.711355 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.711361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.711367 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.711532 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.712025 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.712031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.712033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.712666 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.713934 (+ 1268us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.714179 (+ 245us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.714503 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":289,"thread_start_us":146,"threads_started":1}
I20260805 23:09:55.715837 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.702073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.3.12.124:39325)
0805 23:09:55.702630 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.702639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.702657 (+ 18us) server_negotiation.cc:407] Connection header received
0805 23:09:55.702692 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.702695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.702732 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.702835 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.703752 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.705100 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.706311 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.706704 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.706811 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.710287 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.710291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.710295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.710326 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.713476 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.714217 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.714222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.714224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.714328 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.714704 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.714707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.714709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.714911 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.715094 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.715490 (+ 396us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.715620 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":478,"thread_start_us":111,"threads_started":1}
I20260805 23:09:55.716143 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.703892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.3.12.126:40001)
0805 23:09:55.704019 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.704023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.704313 (+ 290us) server_negotiation.cc:407] Connection header received
0805 23:09:55.704499 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.704505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.704561 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.704623 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.705726 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.706459 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.707888 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.708340 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.708430 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.711253 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.711258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.711264 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.711290 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.713428 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.714277 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.714280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.714282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.714369 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.714713 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.714716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.714718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.714926 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.715000 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.715423 (+ 423us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.715550 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":40}
I20260805 23:09:55.716795 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.703844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.126:40001 (local address 127.0.0.1:51702)
0805 23:09:55.704174 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.704223 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.704395 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.704906 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.704911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.705415 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.705572 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.705581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.706586 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.706593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.707741 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.707753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.708603 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.708605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.708704 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.709356 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.709374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.711098 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.713557 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.713567 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.713573 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.714158 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.714518 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.714521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.714523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.714607 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.715082 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.715088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.715199 (+ 111us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.716395 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.716603 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260805 23:09:55.718118 3845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } has no permanent_uuid. Determining permanent_uuid...
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.125@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
I20260805 23:09:55.721218 3642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef: Bootstrap starting.
I20260805 23:09:55.724066 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.711166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43946)
0805 23:09:55.711290 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.711307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.711473 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.713890 (+ 2417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.713894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.714098 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.714241 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.714250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.715485 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.715489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.716400 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.716413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.717126 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.717129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.717300 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.717840 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.717856 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.719475 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.722187 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.722190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.722192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.722449 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.722750 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.722753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.722755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.722852 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.723461 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.723464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.723650 (+ 186us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.723751 (+ 101us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.723892 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260805 23:09:55.724493 3642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef: Neither blocks nor log segments found. Creating new log.
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
I20260805 23:09:55.724794 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.711843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.3.12.124:39325)
0805 23:09:55.712845 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.712856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.713616 (+ 760us) server_negotiation.cc:407] Connection header received
0805 23:09:55.713650 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.713653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.713715 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.714368 (+ 653us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.714517 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.715326 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.716568 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.716917 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.717117 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.719589 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.719594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.719596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.719619 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.722065 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.722573 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.722576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.722578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.722657 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.723102 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.723106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.723108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.723330 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.724378 (+ 1048us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.724476 (+ 98us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.724602 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":594,"threads_started":1}
I20260805 23:09:55.725466 3642 log.cc:824] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef: Log is configured to *not* fsync() on all Append() calls
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.126@KRBTEST.COM
I20260805 23:09:55.728322 3642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef: No bootstrap required, opened a new log
I20260805 23:09:55.728366 3744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2: Bootstrap starting.
I20260805 23:09:55.730129 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.718870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.3.12.125:34177)
0805 23:09:55.719056 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.719063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.719183 (+ 120us) server_negotiation.cc:407] Connection header received
0805 23:09:55.719263 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.719266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.719311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.719393 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.719853 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.720876 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.722364 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.722800 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.722890 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.725571 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.725575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.725579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.725606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.727375 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.727875 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.727879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.727880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.727980 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.728614 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.728617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.728619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.729324 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.729413 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.729814 (+ 401us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.729954 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260805 23:09:55.730624 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.718629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.125:34177 (local address 127.0.0.1:48870)
0805 23:09:55.719086 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.719102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.719188 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.719424 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.719427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.719587 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.719706 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.719712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.721038 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.721041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.722200 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.722212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.723044 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.723048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.723158 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.723601 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.723617 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.725394 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.727521 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.727524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.727526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.727756 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.728102 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.728105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.728107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.728495 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.729505 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.729509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.729610 (+ 101us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.730308 (+ 698us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.730460 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385}
I20260805 23:09:55.731401 3845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } has no permanent_uuid. Determining permanent_uuid...
I20260805 23:09:55.731477 3744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:55.732101 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.720757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.3.12.126:40001)
0805 23:09:55.720868 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.720874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.720890 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:09:55.720929 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.720933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.720974 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.721052 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.722028 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.722766 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.724137 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.724558 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.724671 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.727881 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.727885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.727888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.727915 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.729872 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.730385 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.730389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.730390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.730490 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.730798 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.730801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.730803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.731011 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.731175 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.731836 (+ 661us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.731967 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260805 23:09:55.733093 3744 log.cc:824] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2: Log is configured to *not* fsync() on all Append() calls
I20260805 23:09:55.734216 3642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } }
I20260805 23:09:55.734493 3642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fbd5b2d1a84595a53023a105b20bef, State: Initialized, Role: FOLLOWER
I20260805 23:09:55.734967 3642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } }
I20260805 23:09:55.736083 3744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2: No bootstrap required, opened a new log
I20260805 23:09:55.736581 3642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: configured and running, proceeding with master startup.
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.12.124@KRBTEST.COM for kudu/127.3.12.124@KRBTEST.COM
Aug 05 23:09:55 dist-test-slave-2d19 krb5kdc[3536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785971395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.12.125@KRBTEST.COM
I20260805 23:09:55.741032 3744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } }
I20260805 23:09:55.741307 3744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0621a8e09c0843559e110224b4c498a2, State: Initialized, Role: FOLLOWER
I20260805 23:09:55.741705 3744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } }
I20260805 23:09:55.742938 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.731853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.12.124:39325 (local address 127.0.0.1:43950)
0805 23:09:55.732428 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0805 23:09:55.732445 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0805 23:09:55.732557 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0805 23:09:55.734457 (+ 1900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0805 23:09:55.734459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0805 23:09:55.734662 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0805 23:09:55.734752 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.734759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.735795 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.735798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.736550 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0805 23:09:55.736561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.737227 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.737230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0805 23:09:55.737300 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.737814 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0805 23:09:55.737830 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0805 23:09:55.739440 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0805 23:09:55.741433 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.741436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.741438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.741668 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.741951 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0805 23:09:55.741955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0805 23:09:55.741956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0805 23:09:55.742064 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0805 23:09:55.742469 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0805 23:09:55.742472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0805 23:09:55.742546 (+ 74us) client_negotiation.cc:769] Sending connection context
0805 23:09:55.742643 (+ 97us) client_negotiation.cc:240] Negotiation successful
0805 23:09:55.742776 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":152,"threads_started":1}
I20260805 23:09:55.743302 3744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:55.743482 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.732238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.3.12.124:39325)
0805 23:09:55.734134 (+ 1896us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.734139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.734154 (+ 15us) server_negotiation.cc:407] Connection header received
0805 23:09:55.734194 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.734199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.734243 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.734347 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.734881 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.735670 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.736679 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.737009 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.737122 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.739578 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.739582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.739584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.739606 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.741309 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.741771 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.741774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.741776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.741856 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.742160 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.742163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.742165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.742350 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.742437 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.743218 (+ 781us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.743337 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1820}
I20260805 23:09:55.744349 3877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.744508 3877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:55.746639 3845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6: Bootstrap starting.
I20260805 23:09:55.742312 3554 master_runner.cc:186] Error getting master registration for 127.3.12.125:34177: OK, instance_id { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" instance_seqno: 1785971395458961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 23:09:55.746909 3554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 23:09:55.748504 3861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.748628 3861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:55.749461 3845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6: Neither blocks nor log segments found. Creating new log.
I20260805 23:09:55.750334 3845 log.cc:824] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6: Log is configured to *not* fsync() on all Append() calls
W20260805 23:09:55.750360 3876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:55.750427 3876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:55.752627 3845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6: No bootstrap required, opened a new log
I20260805 23:09:55.756800 3845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } }
I20260805 23:09:55.757066 3845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401f2557af7d4b58837b4750de85bbd6, State: Initialized, Role: FOLLOWER
I20260805 23:09:55.757416 3845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } }
I20260805 23:09:55.758802 3845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 23:09:55.755858 3653 master_runner.cc:186] Error getting master registration for 127.3.12.124:39325: OK, instance_id { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" instance_seqno: 1785971395689271 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260805 23:09:55.759320 3653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260805 23:09:55.748595 3891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:55.759415 3891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:55.761134 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.733713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.3.12.125:34177)
0805 23:09:55.733824 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.733874 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.733899 (+ 25us) server_negotiation.cc:407] Connection header received
0805 23:09:55.734031 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.734034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.734079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.734129 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.735064 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.735757 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.736834 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.737152 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.737230 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.741404 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.741407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.741410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.741439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.743172 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.743685 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.743689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.743691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.743777 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.759626 (+ 15849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.759630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.759632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.759930 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.760136 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.760748 (+ 612us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.760917 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260805 23:09:55.762041 3900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.762197 3900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:55.767190 3121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260805 23:09:55.768378 3609 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "73fbd5b2d1a84595a53023a105b20bef"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51710
I20260805 23:09:55.768659 3609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 23:09:55.768697 3758 master_runner.cc:263] No leader master found from master 401f2557af7d4b58837b4750de85bbd6
I20260805 23:09:55.768754 3758 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260805 23:09:55.768803 3609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [term 0 FOLLOWER]: Advancing to term 1
W20260805 23:09:55.769995 3915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260805 23:09:55.770079 3915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260805 23:09:55.770370 3609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } }
I20260805 23:09:55.771477 3609 leader_election.cc:288] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [CANDIDATE]: Term 1 election: Requested vote from peers 0621a8e09c0843559e110224b4c498a2 (127.3.12.125:34177), 401f2557af7d4b58837b4750de85bbd6 (127.3.12.124:39325)
I20260805 23:09:55.772310 3707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73fbd5b2d1a84595a53023a105b20bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0621a8e09c0843559e110224b4c498a2"
I20260805 23:09:55.772598 3707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:55.774115 3707 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73fbd5b2d1a84595a53023a105b20bef in term 1.
I20260805 23:09:55.774552 3578 leader_election.cc:302] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0621a8e09c0843559e110224b4c498a2, 73fbd5b2d1a84595a53023a105b20bef; no voters:
I20260805 23:09:55.774860 3861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [term 1 FOLLOWER]: Leader election won for term 1
I20260805 23:09:55.775099 3861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [term 1 LEADER]: Becoming Leader. State: Replica: 73fbd5b2d1a84595a53023a105b20bef, State: Running, Role: LEADER
I20260805 23:09:55.775456 3861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } }
I20260805 23:09:55.777606 3861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73fbd5b2d1a84595a53023a105b20bef. Latest consensus state: current_term: 1 leader_uuid: "73fbd5b2d1a84595a53023a105b20bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.777733 3861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:55.778388 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "73fbd5b2d1a84595a53023a105b20bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "401f2557af7d4b58837b4750de85bbd6"
I20260805 23:09:55.778674 3812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260805 23:09:55.780097 3812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73fbd5b2d1a84595a53023a105b20bef in term 1.
I20260805 23:09:55.784659 3920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 23:09:55.785667 3920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 23:09:55.802927 3812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [term 1 FOLLOWER]: Refusing update from remote peer 73fbd5b2d1a84595a53023a105b20bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:55.803633 3861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:55.808418 3900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73fbd5b2d1a84595a53023a105b20bef. Latest consensus state: current_term: 1 leader_uuid: "73fbd5b2d1a84595a53023a105b20bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.808846 3900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:55.809098 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.788985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.3.12.124:39325)
0805 23:09:55.789120 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.789124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.793858 (+ 4734us) server_negotiation.cc:407] Connection header received
0805 23:09:55.793963 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.793967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.794024 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.794118 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.797012 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.797782 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.800905 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.801268 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.801348 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.805398 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.805402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.805405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.805434 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.807100 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.807602 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.807606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.807608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.807693 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.807989 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.807992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.807994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.808205 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.808349 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.808713 (+ 364us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.808836 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260805 23:09:55.809509 3707 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [term 1 FOLLOWER]: Refusing update from remote peer 73fbd5b2d1a84595a53023a105b20bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 23:09:55.810149 3861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 23:09:55.814388 3877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73fbd5b2d1a84595a53023a105b20bef. Latest consensus state: current_term: 1 leader_uuid: "73fbd5b2d1a84595a53023a105b20bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.814534 3877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:55.819375 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.789331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.3.12.126:40001)
0805 23:09:55.789521 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.789526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.793454 (+ 3928us) server_negotiation.cc:407] Connection header received
0805 23:09:55.794279 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.794282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.794328 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.794386 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.796196 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.797024 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.800123 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.800495 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.800573 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.815329 (+ 14756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.815333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.815336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.815367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.817358 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.817872 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.817879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.817882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.817985 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.818311 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.818315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.818320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.818571 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.818677 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.819055 (+ 378us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.819201 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260805 23:09:55.819576 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:55.788204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.3.12.125:34177)
0805 23:09:55.788354 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.788360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.793263 (+ 4903us) server_negotiation.cc:407] Connection header received
0805 23:09:55.794027 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.794032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.794085 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.794138 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.798212 (+ 4074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.798971 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.801208 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:55.801559 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:55.801638 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:55.815329 (+ 13691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0805 23:09:55.815333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0805 23:09:55.815336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0805 23:09:55.815367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0805 23:09:55.817703 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.818201 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.818205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.818207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.818296 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0805 23:09:55.818614 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0805 23:09:55.818619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0805 23:09:55.818621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0805 23:09:55.818839 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0805 23:09:55.818968 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:55.819296 (+ 328us) server_negotiation.cc:299] Negotiation successful
0805 23:09:55.819428 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260805 23:09:55.823621 3861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73fbd5b2d1a84595a53023a105b20bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.823752 3861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:55.825048 3900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73fbd5b2d1a84595a53023a105b20bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.825210 3900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:55.825500 3936 mvcc.cc:205] Tried to move back new op lower bound from 7315338837164576768 to 7315338837100138496. Current Snapshot: MvccSnapshot[applied={T|T < 7315338837164576768}]
I20260805 23:09:55.825740 3920 catalog_manager.cc:1347] Generated new cluster ID: def69292e4a8412fa3dca0da10e1d7db
I20260805 23:09:55.825809 3920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 23:09:55.826264 3917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73fbd5b2d1a84595a53023a105b20bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.826382 3917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef [sys.catalog]: This master's current role is: LEADER
I20260805 23:09:55.827437 3877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73fbd5b2d1a84595a53023a105b20bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73fbd5b2d1a84595a53023a105b20bef" member_type: VOTER last_known_addr { host: "127.3.12.126" port: 40001 } } peers { permanent_uuid: "0621a8e09c0843559e110224b4c498a2" member_type: VOTER last_known_addr { host: "127.3.12.125" port: 34177 } } peers { permanent_uuid: "401f2557af7d4b58837b4750de85bbd6" member_type: VOTER last_known_addr { host: "127.3.12.124" port: 39325 } } }
I20260805 23:09:55.827569 3877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260805 23:09:55.836356 3939 mvcc.cc:205] Tried to move back new op lower bound from 7315338837164576768 to 7315338837100138496. Current Snapshot: MvccSnapshot[applied={T|T < 7315338837164576768}]
I20260805 23:09:55.854151 3920 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 23:09:55.855453 3920 catalog_manager.cc:1504] Loading token signing keys...
I20260805 23:09:55.867331 3920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73fbd5b2d1a84595a53023a105b20bef: Generated new TSK 0
I20260805 23:09:55.868206 3920 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260805 23:09:55.901086 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.898188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.3.12.125:34177)
0805 23:09:55.898319 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.898324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.899706 (+ 1382us) server_negotiation.cc:407] Connection header received
0805 23:09:55.900141 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.900145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.900199 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.900287 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.900952 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48896: BlockingRecv error: recv got EOF from 127.0.0.1:48896 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260805 23:09:55.902194 3853 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.898999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.3.12.124:39325)
0805 23:09:55.899121 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.899125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.900054 (+ 929us) server_negotiation.cc:407] Connection header received
0805 23:09:55.901442 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.901445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.901495 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.901544 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.902072 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43964: BlockingRecv error: recv got EOF from 127.0.0.1:43964 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260805 23:09:55.904489 3649 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.903058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.3.12.126:40001)
0805 23:09:55.903176 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.903181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.903482 (+ 301us) server_negotiation.cc:407] Connection header received
0805 23:09:55.903653 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.903657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.903711 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.903767 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.904372 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51722: BlockingRecv error: recv got EOF from 127.0.0.1:51722 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260805 23:09:55.909093 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.907492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.3.12.125:34177)
0805 23:09:55.907618 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.907622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.907941 (+ 319us) server_negotiation.cc:407] Connection header received
0805 23:09:55.908170 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.908174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.908248 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.908352 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.908995 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48906: BlockingRecv error: recv got EOF from 127.0.0.1:48906 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260805 23:09:55.914302 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.912657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.3.12.125:34177)
0805 23:09:55.912759 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.912765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.913350 (+ 585us) server_negotiation.cc:407] Connection header received
0805 23:09:55.913546 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.913550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.913601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.913654 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.914186 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48918: BlockingRecv error: recv got EOF from 127.0.0.1:48918 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260805 23:09:55.920361 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.918570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.3.12.125:34177)
0805 23:09:55.918686 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.918691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.919268 (+ 577us) server_negotiation.cc:407] Connection header received
0805 23:09:55.919521 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.919526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.919581 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.919637 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.920232 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48922: BlockingRecv error: recv got EOF from 127.0.0.1:48922 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260805 23:09:55.928445 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.926592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.3.12.125:34177)
0805 23:09:55.926736 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.926741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.927367 (+ 626us) server_negotiation.cc:407] Connection header received
0805 23:09:55.927570 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.927574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.927624 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.927677 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.928233 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48926: BlockingRecv error: recv got EOF from 127.0.0.1:48926 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260805 23:09:55.940843 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.939127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.3.12.125:34177)
0805 23:09:55.939279 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.939285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.939667 (+ 382us) server_negotiation.cc:407] Connection header received
0805 23:09:55.939917 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.939921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.939984 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.940045 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.940719 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48928: BlockingRecv error: recv got EOF from 127.0.0.1:48928 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260805 23:09:55.961853 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.959689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.3.12.125:34177)
0805 23:09:55.959852 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.959859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.960696 (+ 837us) server_negotiation.cc:407] Connection header received
0805 23:09:55.960925 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.960932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.960989 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.961045 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.961720 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48934: BlockingRecv error: recv got EOF from 127.0.0.1:48934 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260805 23:09:55.997591 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:55.996357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.3.12.125:34177)
0805 23:09:55.996491 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:55.996496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:55.996704 (+ 208us) server_negotiation.cc:407] Connection header received
0805 23:09:55.996886 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:55.996888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:55.996939 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:55.997032 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:55.997488 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48946: BlockingRecv error: recv got EOF from 127.0.0.1:48946 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260805 23:09:56.065302 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:56.063836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.3.12.125:34177)
0805 23:09:56.064013 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:56.064019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:56.064204 (+ 185us) server_negotiation.cc:407] Connection header received
0805 23:09:56.064406 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:56.064411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:56.064520 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:56.064624 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:56.065183 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48950: BlockingRecv error: recv got EOF from 127.0.0.1:48950 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260805 23:09:56.197058 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:56.195776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.3.12.125:34177)
0805 23:09:56.195957 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:56.195962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:56.196094 (+ 132us) server_negotiation.cc:407] Connection header received
0805 23:09:56.196305 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:56.196308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:56.196367 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:56.196461 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:56.196944 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48956: BlockingRecv error: recv got EOF from 127.0.0.1:48956 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260805 23:09:56.456759 3746 negotiation.cc:337] Failed RPC negotiation. Trace:
0805 23:09:56.455409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.3.12.125:34177)
0805 23:09:56.455594 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:56.455601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:56.455623 (+ 22us) server_negotiation.cc:407] Connection header received
0805 23:09:56.455836 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:56.455842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:56.455904 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:56.456021 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0805 23:09:56.456594 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43798: BlockingRecv error: recv got EOF from 127.0.0.1:43798 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260805 23:09:56.761046 3891 catalog_manager.cc:1259] Loaded cluster ID: def69292e4a8412fa3dca0da10e1d7db
I20260805 23:09:56.761137 3891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2: loading cluster ID for follower catalog manager: success
I20260805 23:09:56.763046 3891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2: acquiring CA information for follower catalog manager: success
I20260805 23:09:56.764680 3891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0621a8e09c0843559e110224b4c498a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 23:09:56.771287 3915 catalog_manager.cc:1259] Loaded cluster ID: def69292e4a8412fa3dca0da10e1d7db
I20260805 23:09:56.771356 3915 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6: loading cluster ID for follower catalog manager: success
I20260805 23:09:56.773052 3915 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6: acquiring CA information for follower catalog manager: success
I20260805 23:09:56.774127 3915 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 401f2557af7d4b58837b4750de85bbd6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260805 23:09:56.975198 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:56.970991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.3.12.125:34177)
0805 23:09:56.971290 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:56.971296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:56.971338 (+ 42us) server_negotiation.cc:407] Connection header received
0805 23:09:56.971553 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:56.971561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:56.971696 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:56.971822 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:56.972148 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.972742 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.973638 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.973868 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.973992 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:56.974323 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:56.974778 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:56.974926 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:56.974972 (+ 46us) server_negotiation.cc:299] Negotiation successful
0805 23:09:56.975033 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20260805 23:09:56.982978 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:56.978642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.3.12.124:39325)
0805 23:09:56.978924 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:56.978929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:56.978945 (+ 16us) server_negotiation.cc:407] Connection header received
0805 23:09:56.979136 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:56.979138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:56.979265 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:56.979408 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:56.979702 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.980349 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.981304 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.981564 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.981686 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:56.982044 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:56.982504 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:56.982685 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:56.982731 (+ 46us) server_negotiation.cc:299] Negotiation successful
0805 23:09:56.982794 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260805 23:09:56.990566 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:56.986642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.3.12.124:39325)
0805 23:09:56.986771 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:56.986775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:56.986953 (+ 178us) server_negotiation.cc:407] Connection header received
0805 23:09:56.987167 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:56.987173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:56.987226 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:56.987282 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:56.987772 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.988548 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.989451 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.989672 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.989782 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:56.990041 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:56.990134 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:56.990208 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:56.990319 (+ 111us) server_negotiation.cc:299] Negotiation successful
0805 23:09:56.990383 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260805 23:09:56.990875 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:56.986201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.3.12.125:34177)
0805 23:09:56.986353 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:56.986359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:56.986506 (+ 147us) server_negotiation.cc:407] Connection header received
0805 23:09:56.986861 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:56.986867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:56.986964 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:56.987069 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:56.987317 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.988069 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.989341 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.989735 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.989824 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:56.990201 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:56.990280 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:56.990351 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:56.990659 (+ 308us) server_negotiation.cc:299] Negotiation successful
0805 23:09:56.990740 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260805 23:09:56.992017 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0805 23:09:56.986202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.3.12.126:40001)
0805 23:09:56.986534 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0805 23:09:56.986539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0805 23:09:56.986840 (+ 301us) server_negotiation.cc:407] Connection header received
0805 23:09:56.987053 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0805 23:09:56.987058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0805 23:09:56.987235 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0805 23:09:56.987355 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0805 23:09:56.987652 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.988506 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.989833 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0805 23:09:56.990275 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0805 23:09:56.990386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0805 23:09:56.990852 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0805 23:09:56.991510 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0805 23:09:56.991613 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0805 23:09:56.991834 (+ 221us) server_negotiation.cc:299] Negotiation successful
0805 23:09:56.991892 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260805 23:09:56.995637 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3554
I20260805 23:09:57.010200 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3653
I20260805 23:09:57.024219 3121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqJ6txj/build/debug/bin/kudu with pid 3758
2026-08-05T23:09:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3326 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6701 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (301542 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST